Neural network model for generation of compressed haptic actuator signal from audio input

ABSTRACT

A method comprises inputting an audio signal into a machine learning circuit to compress the audio signal into a sequence of actuator signals. The machine learning circuit being trained by: receiving a training set of acoustic signals and pre-processing the training set of acoustic signals into pre-processed audio data. The pre-processed audio data including at least a spectrogram. The training further includes training the machine learning circuit using the pre-processed audio data. The neural network has a cost function based on a reconstruction error and a plurality of constraints. The machine learning circuit generates a sequence of haptic cues corresponding to the audio input. The sequence of haptic cues is transmitted to a plurality of cutaneous actuators to generate a sequence of haptic outputs.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority under 35 U.S.C. § 119(e) to U.S.Provisional Application No. 62/486,337, filed Apr. 17, 2017; U.S.Provisional Application No. 62/552,791, filed Aug. 31, 2017; U.S.Provisional Application No. 62/574,389, filed Oct. 19, 2017; and U.S.Provisional Application No. 62/621,977, filed Jan. 25, 2018, all ofwhich are incorporated by reference herein in their entirety.

BACKGROUND Field of the Disclosure

The present disclosure generally relates to haptic communication, andspecifically to a haptic communication system using cutaneous actuators.

Description of the Related Arts

Haptic, kinesthetic, and cutaneous communication refers to the ways inwhich humans communicate and interact via the sense of touch, andmeasure information arising from physical interaction with theirenvironment. Haptic sense and touch includes information about surfacesand textures, and is a component of communication that is nonverbal andnonvisual. However, haptic communication between users often depends onwhether the users are present in the same location. Haptic communicationbetween users at different locations is often not feasible. Moreover,users having dual-sensory (deaf-blind) impairment may have difficulty incommunication through audio and/or visual means. For example, deaf-blindpeople may miss out on certain information, thereby increasing theirreliance on friends, family or interpreting services if these areavailable. Deaf-blind persons may also find it difficult to use a socialnetworking system that relies on visual and audio cues.

SUMMARY

Embodiments relate to haptic communication and generating the sense oftouch by transmitting haptic signals to generate haptic forces,vibrations, or motions at actuators attached to a user. In embodiments,mechanical stimulation may be used to assist in the creation ofcommunication messages that correspond to a touch lexicon. The hapticcommunication may incorporate cutaneous actuators to transmit thecommunication to a user.

Embodiments also relate to operating cutaneous actuators to producesensation of actions or motions within a part of a user's body bysending a first and second actuator signals. The first actuator signalis sent to a first cutaneous actuator on a first patch of the user'sskin at a first time to cause the first cutaneous actuator to producehaptic outputs on the first patch of the skin. The second actuatorsignal is sent to a second cutaneous actuator on a second patch of theuser's skin at a second time subsequent to the first time to causehaptic outputs on the second patch of the skin.

Embodiments also relate to operating cutaneous actuators to enhancehaptic communication through a path of a receiving user's skin by usingconstructive or destructive interference between haptic outputs.Characteristics of actuator signals are determined and corresponding theactuator signals are generated according to the determinedcharacteristics by signal processing. The generated actuators signalsare sent to the cutaneous actuators. The cutaneous actuators, spacedapart from each other on a patch of skin, generates haptic outputs bythe cutaneous actuators such that the generated haptic outputsconstructively or destructively interfere on the patch of skin.

Embodiments also relate to encoding a speech signal for hapticcommunication using frequency decomposition. A digitized version of thespeech signal is processed. Dominant frequencies in the processed speechsignal are detected to generate dominant frequency information. Thedetected dominant frequencies are sent for generating actuator signals.

Embodiments also relate to a haptic device that comprises a signalgenerator that is configured to receive an input word that is a unit ofa language. The signal generator converts the input word into one ormore phonemes of the input word. The signal generator further convertsthe one or more phonemes into a sequence of actuator signals. Thesequence of actuator signals is formed from a concatenation ofsub-sequences of actuator signals. Each phoneme corresponds to a uniquesub-sequence of actuator signals. The haptic device further comprises atwo dimensional array of cutaneous actuators configured to receive thesequence of actuator signals from the signal generator, each of theactuator signals mapped to a cutaneous actuator of the two dimensionalarray of cutaneous actuators.

Embodiments also relate to a haptic calibration device that comprises asubjective magnitude input device configured to receive a subjectiveforce value indicating a value of a subjective force from a user and aforce location indicating where the subjective force was experienced bythe user. The haptic calibration device further comprises a plurality ofhaptic sensors located on an area of a surface of a user's body, thehaptic sensors configured to detect a force and to output a sensorvoltage value corresponding to the force. The haptic calibration devicecomprises a signal generator configured to receive the subjective forcevalue and the force location from the subjective magnitude input device.The signal generator also receives from at least one of a plurality ofhaptic sensors a sensor voltage value, with the at least one of theplurality of haptic sensors corresponding to the force location. Thesignal generator stores the subjective force value and the correspondingsensor voltage value in a data store. The signal generator generates acalibration curve indicating a correspondence between subjective forcevalues and sensor voltage values for the location where the subjectiveforce was experienced using the data from the data store, wherein thecalibration curve is used to calibrate a haptic feedback device.

Embodiments also relate to a method comprising inputting an audio signalinto a machine learning circuit to compress the audio signal into asequence of actuator signals. The machine learning circuit being trainedby: receiving a training set of acoustic signals and pre-processing thetraining set of acoustic signals into pre-processed audio data. Thepre-processed audio data including at least a spectrogram. The trainingfurther includes training the machine learning circuit using thepre-processed audio data. The neural network has a cost function basedon a reconstruction error and a plurality of constraints. The machinelearning circuit generates a sequence of haptic cues corresponding tothe audio input. The sequence of haptic cues is transmitted to aplurality of cutaneous actuators to generate a sequence of hapticoutputs.

Embodiments also relate to a haptic device that comprises a signalgenerator that is configured to receive an input word that is a unit ofa language written using consonant-vowel pairs. The signal generatorconverts the input word into one or more consonant-vowel pairs of theinput word. The signal generator further converts the one or moreconsonant-vowel pairs into a sequence of actuator signals. The sequenceof actuator signals is formed from a concatenation of sub-sequences ofactuator signals. Each phoneme corresponding to a unique sub-sequence ofactuator signals. The haptic device further comprises a two dimensionalarray of cutaneous actuators configured to receive the sequence ofactuator signals from the signal generator, each of the actuator signalsmapped to a cutaneous actuator of the two dimensional array of cutaneousactuators.

Embodiments also relate to a haptic communication device including anarray of cutaneous actuators to generate haptic sensations correspondingto actuator signals received by the array. The haptic sensations includeat least a first haptic sensation and a second haptic sensation. Thearray includes at least a first cutaneous actuator to begin generatingthe first haptic sensation at a first location on a body of a user at afirst time. A second cutaneous actuator begins generating the secondhaptic sensation at a second location on the body of the user at asecond time later than the first time.

Embodiments also relate to a haptic communication device including oneor more cutaneous actuators to generate haptic vibrations correspondingto actuator signals received by the one or more cutaneous actuators. Adampening member, proximate to a body of a user wearing the hapticcommunication device, focuses the haptic vibrations at one or moredistinct locations on the body. The dampening member has one or morefirst openings, wherein the one or more cutaneous actuators transmit thehaptic vibrations to the one or more distinct locations through the oneor more first openings. A spacing member contacts the dampening memberand is separated from the body by the dampening member. The spacingmember has one or more second openings dimensioned to receive and securethe one or more cutaneous actuators.

Embodiments also relate to a haptic communication system including aspeech signal generator configured to receive speech sounds or a textualmessage and generate speech signals corresponding to the speech soundsor the textual message. An envelope encoder is operably coupled to thespeech signal generator to extract a temporal envelope from the speechsignals. The temporal envelope represents changes in amplitude of thespeech signals. Carrier signals having a periodic waveform aregenerated. Actuator signals are generated by encoding the changes in theamplitude of the speech signals from the temporal envelope into thecarrier signals. One or more cutaneous actuators are operably coupled tothe envelope encoder to generate haptic vibrations representing thespeech sounds or the textual message using the actuator signals.

Embodiments also relate to a haptic communication system including abroadband signal generator to extract parameters from sensor signalsdescribing a message for transmission to a user. Broadband carriersignals are generated by aggregating a plurality of frequencycomponents. Actuator signals are generated by encoding the parametersfrom the sensor signals into the broadband carrier signals. One or morecutaneous actuators are communicatively coupled to the broadband signalgenerator to receive the actuator signals. Haptic vibrations aregenerated corresponding to the actuator signals on a body of the user tocommunicate the message to the user.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the embodiments can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings.

FIG. 1 is an example view of an array of haptic sensors and cutaneousactuators, in accordance with an embodiment.

FIG. 2 is a block diagram of a haptic communication system receiving ahaptic message from a sending user and sending the haptic illusionsignal to a receiving user, in accordance with an embodiment.

FIG. 3 is a block diagram of a haptic communication system converting aspeech message from a source to a haptic signal, in accordance with anembodiment.

FIG. 4 is an example block diagram describing components associated withtraining a machine learning circuit for haptic communication, inaccordance with an embodiment.

FIG. 5 is an example process for haptic communication, in accordancewith an embodiment.

FIG. 6 is an example process for converting speech to hapticcommunication, in accordance with an embodiment.

FIG. 7 is diagram illustrating a wave pattern of a word and acorresponding spectrogram of a word, in accordance with an embodiment.

FIG. 8A is a block diagram illustrating a signal generator for operatingcutaneous actuators, according to an embodiment.

FIG. 8B is a diagram illustrating cutaneous actuators on a part of areceiving user's body to generate illusion of actions or motionsoccurring with the receiving user's body, according to an embodiment.

FIG. 8C is a schematic diagram illustrating three cutaneous actuatorsfor generating sensory illusions of a center point of vibrations in thereceiving user's body moving from one point to another, according to anembodiment.

FIG. 8D illustrates waveforms for generating the sensation of a motionfrom a patch of skin with the cutaneous actuator to another path ofskin, according to one embodiment.

FIGS. 8E through 8G are diagrams illustrating waveforms for generatingvarious sensations, according to embodiments.

FIG. 8H is a flowchart illustrating a method of operating cutaneousactuators to produce illusion of actions or motions occurring within thereceiving user's body, according to one embodiment.

FIG. 9A is a diagram illustrating using auxiliary cutaneous actuators toenhance localization of haptic outputs from a main cutaneous actuator,according to one embodiment.

FIG. 9B is a diagram illustrating cutaneous actuators to enhancelocalization of haptic outputs of multiple cutaneous actuators,according to one embodiment.

FIG. 9C is a diagram illustrating a lattice structure of cutaneousactuators, according to one embodiment.

FIGS. 9D and 9E are diagrams illustrating virtual cutaneous actuatorsusing constructive interference, according to one embodiment.

FIG. 9F is a diagram illustrating waveforms of actuator signals to causethe sensation of a virtual motion, according to one embodiment.

FIG. 9G are diagrams illustrating a virtual motion created by thewaveforms of FIG. 9F, according to one embodiment.

FIG. 9H is a diagram illustrating placement of two cutaneous actuatorsapplied with different frequency modulation, according to oneembodiment.

FIGS. 9I through 9K are diagrams illustrating haptic output patternsdetected by the receiving user, according to embodiments.

FIG. 9L is flowchart illustrating using interference of haptic outputsfrom at least two cutaneous actuators for enhanced haptic communication,according to one embodiment.

FIG. 10A is a block diagram of a speech source performing frequencydecomposition using selective dominant frequencies, according to oneembodiment.

FIG. 10B is a diagram illustrating splitting a digitized version of avoice signal into multiple frames, according to one embodiment.

FIG. 10C is a graph illustrating a result of applying Fast FourierTransform to a portion of the digitized version of the voice signal in aframe, according to one embodiment.

FIG. 10D is a graph illustrating selection of a predetermined number ofdominant frequencies, according to one embodiment.

FIG. 10E is a diagram illustrating placing of actuators on the receivinguser for reproducing the voice signal using frequency decomposition,according to one embodiment.

FIG. 10F is a flowchart illustrating a process of performing hapticcommunication using selective dominant frequencies, according to oneembodiment.

FIG. 11A illustrates an exemplary haptic device with cutaneous actuatorsthat can be used to transmit sequences of haptic illusion signals to auser based on a sequence of input words, according to one embodiment.

FIG. 11B is a block diagram illustrating the components of a system forconverting input words to haptic illusion signals to activate thecutaneous actuators, according to an embodiment.

FIG. 11C illustrates an example of a haptic output for a singlecutaneous actuator from a single haptic illusion signal, according to anembodiment.

FIG. 11D illustrates an example of haptic outputs for two cutaneousactuators from a sequence of two haptic illusion signals, according toan embodiment.

FIG. 11E illustrates an example of haptic outputs of differentdurations, according to an embodiment.

FIG. 11F illustrates an example of haptic outputs of differentfrequencies, according to an embodiment.

FIG. 11G illustrates an example of haptic outputs of differentamplitudes, according to an embodiment.

FIG. 11H illustrates exemplary sequence of haptic outputs for an exampleinput word, according to an embodiment.

FIG. 11I illustrates an alternate arrangement of haptic devices witharrays of cutaneous actuators, according to an embodiment.

FIG. 11J is a flowchart illustrating a method of converting input wordsinto haptic output based on phonemes, according to an embodiment.

FIG. 11K illustrates an exemplary mapping of the phonemes “/p/,” “/b/,”“/t/,” “/v/,” “/f/,”, and “/θ/,” into sequences of haptic outputs,according to an embodiment.

FIG. 11L illustrates examples of haptic outputs for the phonemes “/d/,”“/k/,” “/g/,” “/ð/,” “/s/,” and “/z/,” according to an embodiment.

FIG. 11M illustrates examples of haptic outputs for the phonemes “/∫/,”“/3/,” “/h/,” “/n/,” “/

/,” and “/l/,” according to an embodiment.

FIG. 11N illustrates examples of haptic outputs for the phonemes “/

/,” “/d/,” “/m/,” “/

/,” “/w/,” and “/j/,” according to an embodiment.

FIG. 11O illustrates examples of haptic outputs for the phonemes “/i/,”“/I/,” “/e/,” “/æ/,” and “/ε/,” according to an embodiment.

FIG. 11P illustrates examples of haptic outputs for the phonemes “/

/,” “/Λ/,” “/

/,” “/

/,” and “/u/,” according to an embodiment.

FIG. 11Q illustrates examples of haptic outputs for the phonemes “/

/,” “/

/,” “/aI/,” “/

I/,” and “/a

/,” according to an embodiment.

FIG. 11R is a chart illustrating the consonant phonemes in FIGS.11K-11Q, and the mapping between the location, duration, and type of thehaptic outputs corresponding to each consonant phoneme and the manner ofarticulation and place of articulation of each consonant phoneme,according to an embodiment.

FIG. 11S is a chart illustrating the vowel phonemes in FIGS. 11K-11Q,and the mapping between the location, duration, and type of the hapticoutputs corresponding to each vowel phoneme and the manner ofarticulation and place of articulation of each vowel phoneme, accordingto an embodiment.

FIG. 11T is a chart illustrating the diphthong phonemes in FIG. 11Q, andthe mapping between the location, duration, and type of the hapticoutputs corresponding to each diphthong phoneme and the manner ofarticulation and place of articulation of each diphthong phoneme,according to an embodiment.

FIG. 12A illustrates a haptic device that may be attached to a user'sforearm to calibrate a haptic device based on subjective force,according to an embodiment.

FIG. 12B illustrates detail views of two different examples of thehaptic device of FIG. 12A, according to an embodiment.

FIG. 12C is a block diagram illustrating the components of a system forcalibrating haptic feedback on a per-user basis using subjective forceinputs and voltage readings from sensors, according to an embodiment.

FIG. 12D illustrates a graph representation of the calibration datadescribed with reference to FIG. 12C.

FIG. 12E illustrates an alternative arrangement of generatingcalibration data using cutaneous actuators, according to an embodiment.

FIG. 12F illustrates an example representation of calibration data as agraph using a cutaneous actuator as in FIG. 12E, according to anembodiment.

FIG. 12G is a flowchart illustrating a method for gathering per-usercalibration data for haptic output, according to an embodiment.

FIG. 13A is a block diagram illustrating an unsupervised learning moduleused to train a neural network in compressing an audio input to asequence of haptic cues, according to an embodiment.

FIG. 13B is a block diagram illustrating the use of the neural networkafter it has been trained, according to an embodiment.

FIG. 13C illustrates a conversion using the neural network of a speechsignal into a set of haptic outputs for a set of cutaneous actuators,according to an embodiment.

FIG. 13D is a flowchart illustrating a method for training a machinelearning circuit to generate a set of compressed haptic cues from anacoustic signal, according to an embodiment.

FIG. 14A is a block diagram illustrating the components of a system forconverting the syllables of input words to haptic illusion signals toactivate cutaneous actuators of a haptic device, according to anembodiment.

FIG. 14B illustrates exemplary sequence of haptic syllable hapticoutputs for an example input word, according to an embodiment.

FIG. 14C is a block diagram illustrating the components of aconsonant-vowel pair (Abugida) haptic signal converter for convertingconsonant-vowel pairs of input words to actuator signals, according toan embodiment.

FIG. 14D illustrates an example sequence of haptic C-V pair hapticoutputs for an example input word, according to an embodiment.

FIG. 14E is a flowchart illustrating a method of converting input wordsinto haptic output based on C-V pairs, according to an embodiment.

FIG. 15A is a planar view of an example array of cutaneous actuatorsmounted on a substrate, in accordance with an embodiment.

FIG. 15B is an illustration of example haptic sensations generated atdifferent times, in accordance with an embodiment.

FIG. 15C is a flowchart illustrating an example process of generatinghaptic sensations to create continuous tactile motion along the body ofa user, in accordance with an embodiment.

FIG. 15D is an illustration of example Pacinian and Non Pacinianstimulation, in accordance with an embodiment.

FIG. 15E is a graphical illustration of an example preferred region ofoperation for an array of cutaneous actuators, in accordance with anembodiment.

FIG. 16A is a cross sectional view of an example haptic communicationdevice, in accordance with an embodiment.

FIG. 16B is a perspective view of components of the example hapticcommunication device, in accordance with an embodiment.

FIG. 16C is a perspective view of the example haptic communicationdevice mounted with cutaneous actuators, in accordance with anembodiment.

FIG. 16D is a perspective view of the example haptic communicationdevice including a dampening member, in accordance with an embodiment.

FIG. 16E is a perspective view of an example haptic communication deviceincluding a cutaneous actuator located within a housing, in accordancewith an embodiment

FIG. 16F is a perspective view of the example haptic communicationdevice including a centering member, in accordance with an embodiment.

FIG. 16G is a cross sectional view of the example haptic communicationdevice taken along the X axis of FIG. 16F, in accordance with anembodiment.

FIG. 16H is a perspective view of the example haptic communicationdevice including a rigid substrate, in accordance with an embodiment.

FIG. 16I is a perspective view of a cylindrical end effector, inaccordance with an embodiment.

FIG. 16J is a perspective view of a tapered end effector, in accordancewith an embodiment.

FIG. 16K is a perspective view of a disc end effector, in accordancewith an embodiment.

FIG. 16L is a perspective view of a dome end effector, in accordancewith an embodiment.

FIG. 16M is a perspective view of an example haptic communication deviceincluding a housing that completely encloses a cutaneous actuator, inaccordance with an embodiment.

FIG. 16N is a cross sectional view of the example haptic communicationdevice of FIG. 16M, in accordance with an embodiment.

FIG. 17A is a block diagram illustrating an example haptic communicationsystem for envelope encoding of speech signals and transmission tocutaneous actuators, in accordance with an embodiment.

FIG. 17B illustrates waveform diagrams for an example temporal envelopefor encoding of speech signals and transmission to cutaneous actuators,in accordance with an embodiment.

FIG. 17C illustrates waveform diagrams for example encoded speechsignals for transmission to cutaneous actuators, in accordance with anembodiment.

FIG. 17D is a block diagram of an example envelope encoder for encodingof speech signals and transmission to cutaneous actuators, in accordancewith an embodiment.

FIG. 17E is an illustration of an example process for envelope encodingof speech signals and transmission to cutaneous actuators, in accordancewith an embodiment.

FIG. 18A is a block diagram illustrating an example haptic communicationsystem using broadband actuator signals for transmission to cutaneousactuators, in accordance with an embodiment.

FIG. 18B is waveform diagrams illustrating example broadband carriersignals, in accordance with an embodiment.

FIG. 18C is a block diagram of an example broadband signal generator, inaccordance with an embodiment.

FIG. 18D is an illustration of an example process for hapticcommunication using broadband actuator signals, in accordance with anembodiment.

DETAILED DESCRIPTION

In the following description of embodiments, numerous specific detailsare set forth in order to provide more thorough understanding. However,note that the embodiments may be practiced without one or more of thesespecific details. In other instances, well-known features have not beendescribed in detail to avoid unnecessarily complicating the description.

Embodiments are described herein with reference to the figures wherelike reference numbers indicate identical or functionally similarelements. Also in the figures, the left most digits of each referencenumber corresponds to the figure in which the reference number is firstused.

Haptic Communication System Using Cutaneous Actuators

Embodiments relate to a mode of communication using the sense of touchto convey messages (including speech) to a user's body. A haptic outputin the form of haptic outputs, for example, is generated by cutaneousactuators attached to the user's body. To generate the haptic output, amessage is processed by an algorithm to generate a corresponding hapticillusion signal that is transmitted to a receiving device of the user tooperate the actuators. The generated haptic illusion signals areprocessed into actuator signals for activating the cutaneous actuators.The cutaneous actuators receive the transmitted haptic illusion signalsand transmit the haptic output corresponding to the received hapticillusion signals to a body of a receiving user.

In addition, in some embodiments, the force applied by the cutaneousactuators may be calibrated per-user, such that each user experiences asimilar subjective level of force from the cutaneous actuators for thesame objective level of force generated by the haptic illusion signals.This allows for a consistent experience throughout a user base.Additional details regarding calibration of the cutaneous actuators isdescribed below with reference to FIG. 12.

Development of Lexicon for Haptic Communication

Haptic communication a mode of communication that is alternative oradditional to other modes of communication that is currently widely usedsuch as speech or text. The haptic communication can be agnostic towritten or spoken languages, and can enhance or supplement communicationbetween multiple users through word embedding. The haptic communicationcan also be used to facilitate communication with users with specialneeds such as deaf or blind people.

Words to be included in lexicon for use in haptic communication can beselected on the basis of most frequent use. For transmitting sense ofhaptic social touch from one user to another user, the lexicon mayinclude haptic messages for greeting, parting, giving attention,helping, consoling, calming, pleasant, and reassuring touches. Whenconverting speech to haptic output, most widely used words may beselected and mapped to a predetermined set of haptic symbols, asdescribed below in detail with reference to FIG. 3.

Some words in the lexicon may be associated with a level of affinitybetween the users. For example, consoling touches may be acceptablebetween users who are already familiar with each other. Hence, therelationships between the users can govern which words in the lexicon orhaptic outputs are permissible for transmission between users.

In addition, in some embodiments, phonemes, consonant-vowel pairs, orsyllables, depending on the input words (i.e., input language), may beconverted into sequences of haptic signals and transmitted to cutaneousactuators on a user's body. This results in the user being able tocomprehend the input words without verbal or visual communication. Sucha method may be useful for situations where verbal or visualcommunication is unavailable. Additional details are described belowwith reference to FIGS. 11 and 14. Furthermore, in some embodiments, aneural network may be used to convert an audio signal directly into asequence of haptic signals. The neural network may be trained toconstrain the conversion of the audio signal such that the resultingsequence of haptic signals can be feasibly implemented in a hapticdevice with limited numbers of cutaneous actuators, while still allowingaccurate differentiation of different signals by users. Additionaldetails are described below with reference to FIG. 13.

Example Haptic Sensors and Actuators

FIG. 1 is an example view of an array 100 of haptic sensors 104 andcutaneous actuators 108, in accordance with an embodiment. The hapticsensors 104 is optional and the array 100 may include only the cutaneousactuators 108 for receiving haptic signals. The array 100 may be placedor worn on a body 112 of a receiving user. The haptic sensors 104 andcutaneous actuators 108 may be mounted on a flexible substrate 116configured to be placed on the body 112 of the receiving user. Thesubstrate 116 is made of a flexible material such as plastics (e.g.,polyethylene and polypropylene), rubbers, nylon, synthetics, polymers,etc.

The haptic sensors 104 measure parameters related to social touch on thebody 112 by receiving forces, vibrations, or motions applied to thearray 100. The haptic sensors 104 may be one or more of accelerometers,pressure sensors, and piezoelectric sensors. In one embodiment, thehaptic sensors 104 may receive sensor input from a social touch that maybe represented in dimensions of pressure, temperature, texture, sheerstress, time, and space or a subset thereof. In one embodiment, airborneultrasound transducers may be used as a haptic sensor.

The haptic communication system may include cutaneous actuators 108 thattransmit information related to social touch to the body 112 when a useris sending the information (i.e., the user is now a sending user). Thecutaneous actuators 108 may be positioned on the array 100 to providevibratory feedback when another user sends a haptic communication to thebody 112 of the sending user. In embodiments, one actuator may induce afeedback vibration, while another actuator creates a second vibration tosuppress the first from propagating to unwanted regions of the body 112or array 100 to localize the haptic experience, as described below indetail with reference to FIGS. 9A through 9L.

In embodiments, the cutaneous actuators 108 may be one or more of voicecoils, linear resonant actuators (LRAs), eccentric rotating massactuators, piezoelectric actuators, electroactive polymer actuators,shape memory alloy actuators, pneumatic actuators, microfluidicactuators, and acoustic metasurface actuators. A voice coil includes aformer, collar, and winding and provides motive force to the body 112 ofthe user by the reaction of a magnetic field to current passing throughthe coil. An LRA uses a voice coil and AC input to produce a vibrationwith a frequency and amplitude corresponding to an electrical signal. Aneccentric rotating mass actuator uses the magnetic field of a DC currentto move a small rotating mass to produce lateral vibration. Apiezoelectric actuator produces a vibration corresponding to an ACcurrent applied to it.

In embodiments, the haptic output produced by the cutaneous actuators208 may include information on one or more of pressure, temperature,texture, sheer stress, time, and space.

Example Environment of Haptic Communication System

FIG. 2 is an example block diagram of a haptic communication system 200,in accordance with an embodiment. The haptic communication system 200may include a transmitting device 240, a social networking system 228and a receiving device 268. The haptic communication system 200 mayinclude other components not illustrated in FIG. 2 or the components maybe arranged in a different manner.

A sending user's body 212 may be partially covered by the haptic sensors104. The sending user's body 212 may transmit haptic touches 204 to thehaptic sensors 104. The haptic sensors 104 may convert the haptictouches 204 to sensor signals 207. The sensor signals 207 may includecurrent, voltage, pressure, some other type of sensor signal, or acombination thereof.

The transmitting device 240 is coupled to the haptic sensors 104. Thetransmitting device 240 includes, among other components, a sensorinterface circuit 220 to receive sensor signals 207 from haptic sensors104 and sends out a signal 256 for conversion to a haptic illusionsignal 202. Alternatively or in addition to receiving the sensor signals207, the transmitting device 240 may send a message in text form (e.g.received via an input device 296 such as a keyboard) for conversion tothe haptic illusion signal 202. The transmitting device 240 sends thesignal 256 based on the sensor signals 207 or message to the network260, which may include any combination of local area and/or wide areanetworks, using both wired and/or wireless communication systems.

In one embodiment, the transmitting device 240 may include a signalgenerator to generate and transmit actuator signals (based on the sensorsignals 207) to an array of cutaneous actuators (illustrated anddescribed in detail below with reference to FIGS. 15A through 15E)placed on a receiving user's body 280 to cause the array to createcontinuous tactile motion along the body 280 of the receiving user. Theactuator signals may correspond to words of a social touch lexicon. Theactuator signals may include information on pressure, temperature,texture, sheer stress, time, and space of a physical touch perceived bythe receiving user's body 280. These parameters (e.g., pressure) may beused by the array to generate haptic outputs corresponding to theparameters received by the array. For example, the parameters may betranslated into a duration time, frequency, or amplitude of the hapticoutputs. Alternately, the transmitting device 240 may send actuatorsignals to the social networking system 228 to adjust the actuatorsignals before they are transmitted to the body 280 of the receivinguser.

The social networking system 228 communicates with haptic devices, e.g.,haptic sensor 104, used by users of the social networking system 228over the network 260. The social networking system 228 may include aprocessor circuit 232 to apply a conversion algorithm to the signals 256to generate haptic illusion signals 202. The conversion algorithm may beembodied as a transfer function (also known as a transfer curve) that isa mathematical representation for fitting or describing the mapping of asignal 256 to the haptic illusion signals 202. The transfer function maybe a representation in terms of spatial or temporal frequency, of therelation between the input and output of the signals 256 to hapticillusion signals 202 mapping system with zero initial conditions andzero-point equilibrium. In one embodiment, the social networking system228 may include a signal generator to generate actuator signals to causean array of cutaneous actuators placed on the receiving user's body 280to create continuous tactile motion along the body of the user 280.

In one embodiment, the haptic illusion signals 202 generated by thesocial networking system 228 may be generated by combining generichaptic illusion signals with personalized haptic illusion signalscustomized to the sending user. The generic haptic illusion signals maycorrespond to predetermined the social touch lexicon such as “Greeting,”“Parting,” “Consoling,” etc. The haptic illusion signals 202 may includeone or more of aggregate values based on the signals 256, a shape of awave of the received sensor signals 207, a frequency-domainrepresentation of the signals 256, and a time-domain sample of thereceived signals 256. The processor circuit 232 may convert the signals256 from the time or space domain to the frequency domain, e.g., using aFourier transform to create a frequency-domain representation. Theprocessor circuit 232 may performing time-domain sampling of the signals256. Time-domain sampling reduces the signals 256 to a set ofdiscrete-time samples, such as a set of values at a point in time and/orspace.

In one embodiment, the processor circuit 232 may include a featureextraction component 236 operatively communicate with the sensorinterface circuit 220 and extract features 244 from the received signals256, as illustrated and described in detail below with respect to FIG.4. The feature extraction component 236 may extract the features 244 byone or more of creating a frequency-domain representation of thereceived signals 256, and performing time-domain sampling of the signals256.

The processor circuit 232 may include a machine learning circuit 242communicating with the feature extraction component 236 to determine atouch signature of the sending user based on the extracted features 244and automatically encode the touch signature into the haptic illusionsignals 202. The touch signatures may be represented as a vector ofvalues for each defined word of a social touch lexicon. For qualitativemodels, this touch signature vector may include values 0 or 1.

In one embodiment, the feature extraction component 236 extractsfeatures from a user profile of the receiving user. The machine learningcircuit 242 may further determine a score for the generated hapticillusion signals 202 based on the extracted features. The score mayindicative of a likelihood of the receiving user expressing a preferencefor the haptic illusion signals 202 based on the user profile.

In one embodiment, the processor circuit 232 determines an affinitybetween the sending user and the receiving user, and modifies the hapticillusion signals 202 based on the determined affinity. For example, theprocessor circuit 232 may alter generic haptic illusion signals for“Greeting” to be reflective of a more intimate relationship between thefirst and receiving users if the affinity between them reflects thatthey are partners. In one embodiment, the processor circuit 232retrieves a user profile of the receiving user, and modifies the hapticillusion signals 202 based on the retrieved user profile. For example,the processor circuit 232 may alter the haptic illusion signals 202 tobe gentler (i.e., weaker) if the receiving user is of the female gender.

In embodiments, the processor circuit 232 may generate the haptic symbolset, where a distance between each haptic symbol of the haptic symbolset and each other haptic symbol of the haptic symbol set is greaterthan a threshold. The distance described herein may indicate, forexample, spatial distances between the actuators operated by the hapticillusions signal 202, a timing difference between vibrations of the sameor different actuators. The benefits and advantages of this method arethat the haptic symbols of the resulting haptic symbol set aredistinguishable from each other. The receiving user can thus more easilycomprehend the haptic communication.

In one embodiment, the distance between each haptic symbol of the hapticsymbol set and each other haptic symbol of the haptic symbol set may bedetermined by one or more of a Minkowski distance, a Mahalanobisdistance, a Matsushita distance, a chi-squared distance, a Hammingdistance, a cosine similarity, a dot product, and a Grassmann distance.

The haptic communication system 200 may further generate an implicitlearning program based on the haptic symbols and the received speechsignals, where the implicit learning program is immersive,non-intentional, age-independent, IQ-independent, and subconscious. Thebenefits and advantages of this approach are that the receiving user maymore easily learn the haptic symbol set and the particular language ofhaptic communication that it represents.

In one embodiment, the social networking system 228 includes an edgestore 248 that stores information describing connections between userson the social networking system 228 as edges. Some edges may be definedby users, allowing users to specify their relationships with otherusers. For example, users may generate edges with other users thatparallel the users' real-life relationships, such as friends,co-workers, partners, and so forth. Other edges are generated when userssends haptic communication, such as “Greeting,” “Parting,” “Consoling,”etc., speech signals 216, or social networking emoticons, e.g., “like,”“love,” “haha,” “wow,” “sad,” “angry,” etc. Other edges are generatedwhen users interact with content items in the social networking system228, such as expressing interest in a page on the social networkingsystem, sharing a link with other users of the social networking system,and commenting on posts made by other users of the social networkingsystem.

The edge store 248 also stores information about edges, such as affinityscores for content items, interests, and other users. Affinity scores,or “affinities,” may be computed by the social networking system 228over time to approximate a user's affinity for a type of hapticcommunication emotion associated with user interactions, an contentitem, interest, and other users in the social networking system 228based on the actions performed by the user. For example, the edge store248 may determine a user's affinity for the words “Greeting,” “Parting,”“Consoling,” etc., as the number of times the user sent these words toanother user. Computation of affinity is further described in U.S. Pat.No. 8,402,094, which is hereby incorporated by reference in itsentirety.

Each user of the social networking system 228 may be associated with auser profile, which is stored in the user profile store 284. A userprofile includes declarative information about the user that wasexplicitly shared by the user and may also include profile informationinferred by the social networking system 228. In one embodiment, a userprofile includes multiple data fields, each describing one or moreattributes of the corresponding user of the social networking system228. Examples of information stored in a user profile includebiographic, demographic, and other types of descriptive information,such as user profile images, work experience, educational history,gender, hobbies or preferences, location and the like. A user profilemay also store other information provided by the user, for example,images or videos. In certain embodiments, user profile images of usersmay be tagged with identification information of users of the socialnetworking system 228.

In embodiments, the user profile store 284 may include, for each user,an avatar, a screenname, and the user's real name. An avatar is an iconor figure representing a particular user in computer games, Internetforums, social networking systems, etc. A screenname is a uniquesequence of characters that a user may choose to use for identificationpurposes when interacting with others online, as in computer games,instant messaging, forums, and via the social networking system 228.

The social networking system 228 may continuously update the userprofile for a user with the geolocation of the user's haptic sensor 104.A user's geolocation may be determined by the social networking system228 based on information sent by a user's haptic sensor 104's GPS chipand satellite data, which mapping services can map. When a GPS signal isunavailable, the social networking system 228 may use information fromcell towers to triangulate a user's haptic sensor 104's position or GPSand cell site triangulation (and in some instances, local Wi-Finetworks) in combination to zero in on the location of the user's hapticsensor 104; this arrangement is called Assisted GPS (A-GPS). The socialnetworking system 228 may also determine the geolocation distancebetween two user's haptic sensors using the Haversine formula tocalculate the great-circle distance between two points, as a straightline distance between the two client devices, which are associated withgeolocation coordinates in terms of latitude and longitude, etc.

User interaction store 288 may be used by the social networking system228 to track haptic communication on the social networking system 228,as well as interactions on third party systems that communicateinformation to the social networking system 228. Users may interact withother users on the social networking system 228, and informationdescribing these interactions is stored in the user interaction store288. Examples of interactions using haptic communication may include:commenting on posts, sharing links, and checking-in to physicallocations via a mobile device, accessing content items, etc. Additionalexamples of interactions on the social networking system 228 that may beincluded in the user interaction store 288 are commenting on a photoalbum, communicating with a user, establishing a connection with ancontent item, joining an event to a calendar, joining a group, creatingan event, authorizing an application, using an application, expressing apreference for an content item (“liking” the content item) and engagingin a transaction.

The user interaction store 288 may store information corresponding tohaptic symbols or words, where each word includes informationidentifying an emotion type. Additionally, the user interaction store288 may record a user's interactions with other applications operatingon the social networking system 228. In some embodiments, data from theuser interaction store 288 is used to infer interests or preferences ofa user, augmenting the interests included in the user's user profile andallowing a more complete understanding of user preferences.

The user interaction manager 292 receives communications about userhaptic communication internal to and/or external to the socialnetworking system 228, populating the user interaction store 288 withinformation about user interactions. Interactions received by the userinteraction manager 292 may include expressing an emotional preferencefor words in a haptic lexicon. In addition, a number of actions mayinvolve one or more particular users, so these actions are associatedwith those users as well and stored in the user interaction store 288.The user interaction manager 292 may determine the time a user performedan interaction by a timestamp in the haptic illusion signals 202 orsensor signals 207 representing the interaction.

The transmitting device 240 includes input devices 296 such as akeyboard or pointing device to receive, such as a selection of a certainword from the lexicon. For example, a user may type in a word such as“Greeting,” “Parting,” “Consoling,” etc., using the input device 296 andthe social networking system 228 may generate haptic illusion signals202 corresponding to these words.

The haptic communication system 200 may include a receiving device 268that receives haptic illusion signals 202 from the social networkingsystem 228 over the network 260 and transmit the haptic illusion signals202 to cutaneous actuators 208. In one embodiment, the receiving device268 may include an interface circuit that the haptic illusion signals202 from the network 260. The receiving device 268 may generate actuatorsignals 272 corresponding to the haptic illusion signals 202.

The cutaneous actuators 208 receive the transmitted haptic illusionsignals 202 and transmit haptic output 276 (e.g., vibrations)corresponding to the received haptic illusion signals 202 to a body 280of a receiving user. In one embodiment, the cutaneous actuators 208transmit the haptic output 276 to C tactile (CT) afferent nerve fibersof the body 280 of the receiving user. In the peripheral nervous system(PNS), a CT afferent nerve fiber is the axon of a sensory neuron. Itcarries an action potential from the sensory neuron toward the centralnervous system (CNS). In one embodiment, the receiving device 268transmits the actuator signals 272 to the cutaneous actuators 208. Thecutaneous actuators 208 receive the actuator signals 272 and transmithaptic outputs 276 corresponding to the actuator signals 272 to the body280 of the receiving user.

An external authorization server or an authorization server internal tothe social networking system 228 enforces one or more privacy settingsof the users of the social networking system 228. A privacy setting of auser determines how particular information associated with a user can beshared, and may be stored in the user profile of a user in the userprofile store 284 or stored in the authorization server and associatedwith a user profile. In one embodiment, a privacy setting specifiesparticular information associated with a user and identifies the entityor entities with whom the specified information may be shared. Examplesof entities with which information can be shared may include otherusers, applications, third party systems or any entity that canpotentially access the information. Examples of information that can beshared by a user include user profile information like profile photo,phone numbers associated with the user, user's connections, actionstaken by the user such as adding a connection, changing user profileinformation and the like.

The privacy setting specification may be provided at different levels ofgranularity. In one embodiment, a privacy setting may identify specificinformation to be shared with other users. For example, the privacysetting identifies a work phone number or a specific set of relatedinformation, such as, personal information including profile photo, homephone number, and status. Alternatively, the privacy setting may applyto all the information associated with the user. Specification of theset of entities that can access particular information may also bespecified at various levels of granularity. Various sets of entitieswith which information can be shared may include, for example, all usersconnected to the user, a set of users connected to the user, additionalusers connected to users connected to the user all applications, allthird party systems, specific third party systems, or all externalsystems.

One embodiment uses an enumeration of entities to specify the entitiesallowed to access identified information or to identify types ofinformation presented to different entities. For example, the user mayspecify types of actions that are communicated to other users orcommunicated to a specified group of users. Alternatively, the user mayspecify types of actions or other information that is not published orpresented to other users.

The authorization server includes logic to determine if certaininformation associated with a user can be accessed by a user's friends,third-party system and/or other applications and entities. For example,a third-party system that attempts to access a user's comment about auniform resource locator (URL) associated with the third-party systemmust get authorization from the authorization server to accessinformation associated with the user. Based on the user's privacysettings, the authorization server determines if another user, athird-party system, an application or another entity is allowed toaccess information associated with the user, including information aboutactions taken by the user. For example, the authorization server uses auser's privacy setting to determine if the user's comment about a URLassociated with the third-party system can be presented to thethird-party system or can be presented to another user. This enables auser's privacy setting to specify which other users, or other entities,are allowed to receive data about the user's actions or other dataassociated with the user.

Although the embodiment of FIG. 2 was described as the transfer functionbeing applied in the social networking system 228, the transfer functioncan be executed at the transmitting device 240 to generate the hapticillusion signals 202 at the transmitting device 240. In such embodiment,the further processing by the social networking system 228 may not beperformed. Alternatively, the transmitting device 240 may send thesensor signals 207 to the receiving device 268 where the transferfunction can be executed to generate the actuator signals 272.

Moreover, although FIG. 2 illustrates a one-way haptic communicationwhere the sending user sends a haptic signal to the receiving user, thehaptic communication can be bidirectional. Further, a sending user canbroadcast the haptic signal to a plurality of receiving users instead ofa single user.

Example System for Converting Speech to Haptic Signals

FIG. 3 is a block diagram of a haptic communication system 300converting a speech message from a source 318 to a haptic signal, inaccordance with an embodiment. The haptic communication system 300decomposes speech signals into speech subcomponents and converts thespeech subcomponents to distinct haptic symbols mapped to the speechsubcomponents. The speech source 318 may process signals from a source(e.g., microphone) to generate speech signals 216. The speech signals216 may be sent directly to the receiving device 268 or via the socialnetworking system 228. For this purpose, the speech source 318 mayinclude a microphone to detect voice signals from a sending user and/ora media storage that stores recordings of voice messages, as describedbelow with reference to FIG. 10A.

Similar to the embodiment described above with reference to FIG. 2, thehaptic communication system 300 includes the receiving device 268 andthe social networking system. The operations and functions of thereceiving device 268 and the actuators 208 in FIG. 2 are substantiallythe same as those of FIG. 3 except that the haptic illusion signals 202represent a speech message, and therefore, detailed description thereofis omitted herein for the sake of brevity.

The processor circuit 232 may generate the haptic symbol set bygenerating unique haptic symbols corresponding to the speechsubcomponents. The processor circuit 232 may identify a subset of theunique haptic symbols by characterizing a degree of similarity betweeneach haptic symbol of the subset and each other haptic symbol of thesubset. The processor circuit 232 may perform clustering analysis toidentify groups of haptic symbols having a distance between members ofthe groups greater than a threshold. The processor circuit 232 mayassociate the subset with the haptic symbol set.

In one embodiment, the processor circuit 232 of the social networkingsystem divides the compressed spectrum of the speech signals 216 amongmultiple actuators 208 attached to the receiving user's body. As anexample, vowels are recognized by the presence of significant lowfrequency energy and the localization of the energy in multiple discreteenergy bands (also known as ‘formants’) as illustrated by FIG. 7, whichshows the raw recording of the word ‘facebook’ and the spectrogram. Thespectrogram shows the component frequencies as a function of time. Ascan be seen from the spectrogram the vowels ‘A’ and ‘oo’ have multiplebands of relatively high energy in the low frequency region. In contrast‘f’, ‘c’, ‘b’, and ‘k’ are brief, low energy, and lack the localizedenergy band structure that vowels have. Detection of a low energy signalthat lacks the frequency structure of vowels will trigger the playbackspeed to slow, allowing the user more time to appreciate the subtlerpattern of the consonant.

In one embodiment, the cutaneous actuators 208 transmit the hapticoutputs 276 by transmitting haptic output corresponding to vowels of thereceived speech signals 216 at a first speed. The cutaneous actuators208 may transmit haptic output corresponding to consonants of thereceived speech signals 216 at a second speed lower than the firstspeed. The advantages and benefits of this approach are that vowels arerelatively easy to recognize using automated techniques and by humans ina haptic representation of speech. However, consonants are shorter induration and have ambiguous spectral signatures. By slowing the dynamicsof the haptic presentation during consonants, relative to the speed atwhich vowels are displayed, the accuracy of perception will increasewith less degradation of speed compared to uniform slowing.

In one embodiment, the processor circuit 232 slows the playback ofactuator signals 272 while preserving the component frequencies at theiroriginal component values. In another embodiment the frequencies aretransformed to lower values according to the time/frequency scalingapproach described in the next paragraph under ‘Frequency compression’.

In one embodiment, the speech spectrum of the received speech signals216 may be reduced to a narrower range, while maintaining thedistribution of sound waves and their inter-relationships in the speechsignals 216. The speech source 318 may encode the speech signals 216using a slower sampling rate. The sensor interface circuit 220 maymodify characteristics of the speech signals 216 such as rhythmic andtemporal patterns, pitch and duration of segmental elements. The speechsource 318 may monotonically compress the short time spectrum of thespeech signals 216, without changing the pitch.

In one embodiment, the processor circuit 232 of the social networkingsystem 228 may split the speech signals into speech subcomponents. Thespeech subcomponents may include one or more of phonemes of the receivedspeech signals 216, frequencies of the received speech signals 216,formants of the received speech signals 216, and semantics of thereceived speech signals 216. A phoneme is any of the perceptuallydistinct units of sound in a specified language that distinguish oneword from another, for example p, b, d, and t in the English words pad,pat, bad, and bat. A formant refers to each of several prominent bandsof frequency that determine the phonetic quality of vowels in the speechsignals 216. Semantics may include logical aspects of meaning, such assense, reference, implication, and logical form, lexical semantics (wordrelations), or conceptual semantics (the cognitive structure ofmeaning).

In one embodiment, when the speech subcomponents are frequencies of thespeech signals, the instantaneous power in each frequency banddetermines the amplitude of respective actuators.

In one embodiment, the processor circuit 232 splits the received signals216 by phoneme decomposition or Mel frequency cepstral coefficients(MFCC) to transform instantaneous power of frequency bands of thereceived speech signals. For example, the processor circuit 232 mayidentify phonemes from the signals 216 by computing the MFCC. Themachine learning circuit 242, in the form of a feed forward neuralnetwork (FFNN) may be trained by back propagation procedure foridentifying the phonemes. The extracted MFCC coefficients may then beused as input to a machine learning classifier. When the speechsubcomponents are frequencies of the speech signals, the instantaneouspower in each frequency band is determined using mel frequency ceptralcoefficients.

The processor circuit 232 may map the speech subcomponents to hapticsymbols of a haptic symbol set. The haptic symbol set may correspond towords of a generic social touch lexicon. The processor circuit 232 mayconvert the haptic symbols into the haptic illusion signals 202 oractuator signals. In one embodiment, the processor circuit 232 convertsthe haptic symbols by using a two-dimensional frequency mapping of firstand second formants of the speech subcomponents to determine preferredlocations of the cutaneous actuators 208 on the body of a receivinguser. In other embodiments other pairs of formants may be used, e.g.,the first and the third formants. In still other embodiments, thedifferences between formants define a point on a 2d map, e.g., f2−f1 andf3−f2 are mapped to a physical location of different actuators 208,where f1, f2, and f3 represent the frequencies of the first, second andthird formants, respectively.

In one or more embodiments, consonants can be mapped separately, e.g.,by using a one-to-one mapping between specific consonants and anactuator at a specific location of the receiving user's body.

In other embodiments, features of speech articulation are encoded foroperating certain actuators rather than encoding the features of thespeech signal. The features of speech articulation can include, forexample, the location of occlusion during oral occlusive sounds: lips([p], [b]), tongue blade ([t], [d]), tongue body ([k], [g]), or glottis([

]).

In one embodiment, the machine learning circuit 242 may determine thespeech subcomponents based on the extracted features 544 and generatethe haptic symbols corresponding to the determined speech subcomponents.

Although FIG. 3 describes embodiments where the haptic illusion signals202 is generated at the social networking system 228, the processing forgenerating the haptic illusion signals 202 can be performed at thespeech source or the receiving device 268. For example, the receivingdevice 268 may be a portable device that includes a microphone tocapture speech signals of other people around the user and a processorfor converting the captured speech signals to haptic outputs.

Additional details regarding converting audio and speech to hapticsignals is provided below with reference to FIGS. 11 and 14.

Example Block Diagram for Machine Learning

FIG. 4 is an example block diagram describing components associated withtraining a machine learning circuit 242 for haptic communication, inaccordance with an embodiment. The components associated with themachine learning circuit 242 includes a feature extraction component236, a touch signatures store 412 and a speech subcomponents store 416.The components illustrated in FIG. 4 may be distributed across differentdevices. Some of the processes associated with these components may beexecuted in parallel or sequentially. Alternatively, some processes maybe executed in a pipelined fashion such that execution of a process isstarted before the execution of a previous process.

The feature extraction component 236 receives the signals 216, 256 andextracts features 408 a, 408 b, etc., from the signals 216, 256. Thefeatures 408 a, 808 b, etc., facilitate training of the machine learningcircuit 242. In one embodiment, redundant input data in the signals 216,256 such as the repetitiveness of signals or speech patterns may betransformed into the reduced set of features 408. The extracted features408 contain the relevant information from the signals 216, 256 such thatthe machine learning circuit 242 is trained by using this reducedrepresentation instead of the complete initial data. The features 408corresponding to the signals 216, 256 are used for training the machinelearning circuit 242 based on known touch signatures stored in the touchsignatures store 412 and known speech subcomponents stored in the speechsubcomponents store 416 that correspond to those features.

The touch signatures store 412 stores known touch signatures determinedduring an earlier calibration and training phase. The known touchsignatures may correspond to a single user or a group of users. Thetouch signatures store 412 may be organized as a database, table, file,etc., stored on one or more of removable or non-removable memory cards,computer hard drives, etc. In one embodiment, the touch signatures store412 includes multiple data fields, each describing one or moreattributes of sensor signals 207. In one embodiment, the speechsubcomponents store 416 is used to store known speech subcomponentsdetermined during an earlier calibration and training phase. The knownspeech subcomponents may correspond to a single user or a group ofusers. The speech subcomponents store 416 may be organized as adatabase, table, file, etc., stored on one or more of removable ornon-removable memory cards, computer hard drives, etc. In oneembodiment, the speech subcomponents store 416 includes multiple datafields, each describing one or more attributes of the speech signals216.

The features 408 may include a feature 408 a describing afrequency-domain representation of the signals 216, 256. Extracting thefeature 408 a may include creating a frequency-domain representation ofthe signals 216, 256. The features 408 may include a feature 408 bdescribing a time-domain representation of the signals 216, 256.Extracting the feature 408 b may include performing time-domain samplingof the signals 216, 256. The features 408 may include a feature 408 cdescribing aggregate values based on the signals 216, 256. The features408 may include a feature 408 d describing a shape of a wave of thesignals 216, 256. The features 408 may include a feature 408 edescribing phonemes of the speech signals 216.

The machine learning circuit 242 functions as an autoencoder and istrained using training sets including information from the touchsignatures store 412 and the speech subcomponents store 416. The touchsignatures store 412 may store associations between known touchsignatures for the touch lexicon. In embodiments, the machine learningcircuit 242 is thereby configured to apply the transfer function to thesignals 216, 256 by extracting features 408 from the signals 216, 256,determining a touch signature of the sending user based on the extractedfeatures 408, and generating the haptic illusion signals 202corresponding to the determined touch signature. In one embodiment, themachine learning circuit 242 determines speech subcomponents based onthe extracted features 408 and generates haptic symbols corresponding tothe determined speech subcomponents.

As described above, in some embodiments coarsely quantized signals areneeded. In order to accomplish this the bottleneck layer (which, aftertraining, will be what drives the haptic actuators) will have units thathave coarsely quantized outputs, e.g., binary or trinary. In otherembodiments the hidden layer will use a floating point representationwith the output of that layer appropriately transformed to generatehaptic gestures that are distinguishable by the user.

In one embodiment, the machine learning circuit 242 may transmit thehaptic illusion signals 202 to the receiving device 268 if a likelihoodthat the haptic illusion signals 202 correspond to the signals 216, 256exceeds a threshold. The likelihood may be indicative of a probabilitythat the features 408 have a particular Boolean property or an estimatedvalue of a scalar property. As part of the training of the machinelearning model 242, the process may form a training set of features 408,touch signatures, and speech subcomponents by identifying a positivetraining set of features that have been determined to have the propertyin question, and, in some embodiments, forms a negative training set offeatures that lack the property in question. In one embodiment, themachine learning training applies dimensionality reduction (e.g., vialinear discriminant analysis (LDA), principle component analysis (PCA),or the like) to reduce the amount of data in the features 408 to asmaller, more representative set of data.

The process uses machine learning to train the machine learning circuit242 with the features of the positive training set and the negativetraining set serving as the inputs. Different machine learningtechniques such as linear support vector machine (linear SVM), boostingfor other algorithms (e.g., AdaBoost), neural networks, logisticregression, naïve Bayes, memory-based learning, random forests, baggedtrees, decision trees, boosted trees, or boosted stumps may be used indifferent embodiments. The machine learning circuit 242, when applied tonew features extracted from the signals 216, 256, outputs an indicationof whether the signals 216, 256 have the property in question, such as aBoolean yes/no estimate, or a scalar value representing a probability.

In some embodiments, three datasets are used. Training for the machinelearning circuit 242 is performed using the first dataset. The accuracyof the machine learning circuit 242 is tested using the second dataset.The third dataset, known as a validation set, may be formed ofadditional features, other than those in the training sets, which havealready been determined to have or to lack the property in question. Theprocess applies the trained machine learning circuit 242 to the featuresof the validation set to quantify the accuracy of the machine learningcircuit 242. Common metrics applied in accuracy measurement include:Precision (P)=True Positives (TP)/(TP+False Positives (FP)) and Recall(R)=TP/(TP+False Negatives (FN)), where Precision refers to how manytouch signatures the machine learning circuit 242 correctly predicted(TP) out of the total it predicted (TP+FP), and Recall refers to howmany touch signatures the machine learning circuit 242 correctlypredicted (TP) out of the total number of features that did have theproperty in question (TP+FN). In one embodiment, the process iterativelyre-trains the machine learning circuit 242 until the occurrence of astopping condition, such as the accuracy measurement indication that themodel is sufficiently accurate, or a number of training rounds havingtaken place.

Example Process for Haptic Communication

FIG. 5 is an example process for haptic communication, in accordancewith an embodiment. The social networking system 228 detects 500 anaction invoking haptic communication. For example, the social networkingsystem 228 may receive sensor signals 207 from haptic sensor 104detecting haptic input from a body 212 of a sending user. The hapticsensor 104 generate the sensor signals 207 corresponding to the receivedsensor input 204. Alternatively or in addition, the social networkingsystem 228 may receive messages from input devices 296 such as thekeyboard.

The processor circuit 232 applies 504 a transfer function to thereceived sensor signals 207 or other input to generate haptic illusionsignals 202. The social networking system 228 transmits 508 thegenerated haptic illusion signals 202 to cutaneous actuators 208. Thecutaneous actuators 208 receive the transmitted haptic illusion signals202 and transmit haptic outputs 276 corresponding to the received hapticillusion signals 202 to a body 280 of a receiving user.

Example Process for Converting Speech to Haptic Communication

FIG. 6 is an example process for converting speech to hapticcommunication, in accordance with an embodiment. The processor circuit232 receives 610 speech signals 216 from the speech source 318 (e.g.,via a microphone). The processor circuit 232 splits 614 the receivedsignals 216, 256 into speech subcomponents. The processor circuit 232maps 618 the speech subcomponents to haptic symbols of a haptic symbolset. The processor circuit 232 converts 622 the haptic symbols intoactuator signals. The receiving device 268 transmits 626 the actuatorsignals to cutaneous actuators 208. The cutaneous actuators 208 receivethe actuator signals and transmit haptic outputs 276 corresponding tothe actuator signals to a body 280 of a user.

In one embodiment, the haptic communication may be part of a multi-modalcommunication. The social networking system 228 may, for example, selectone or more modes of communication among text messages, audio signalsand haptic communication for transmission to a user based on variousfactors such as availability of actuators on the user, affinity betweena sending user and a receiving user, and preference of the users. Also,multiple modes of communication can be combined to transmit a message inan effective manner. For example, a message may be transmitted using acombination of audio signals, haptic output and a visual signal. Toincrease the efficiency of the multimodal communication, coding theorysuch as Huffman coding may be used to design a combination of messageelements in different modes of communication.

Additional details regarding converting speech to haptic communicationsis described below with reference to FIGS. 11 and 14.

Generating Inside the Body Illusions

Embodiments also relate to operating multiple cutaneous actuators toprovide the sensation of motions or actions occurring within the body. Apart of receiving user's body (e.g., limb or head) is placed between thecutaneous actuators. The cutaneous actuators are operated with atemporal relationship or relative amplitudes, causing the illusion ofmotions or actions occurring inside the body part. By differing temporalrelationship or relative amplitudes between the haptic output (e.g.,vibrations) generated by the cutaneous actuators, differing sensationsof actions or motions within the body can be generated.

FIG. 8A is a block diagram illustrating a signal generator 800 foroperating cutaneous actuators 802A through 802N (hereinaftercollectively referred to as “cutaneous actuators 802”), according to anembodiment. The signal generator 800 may be part of the receiving device268 or it can be a stand-alone device that generates actuator signals812A through 812N (hereinafter collectively referred to as “actuatorsignals 812”) for transmitting to the cutaneous actuators 802. When thesignal generator 800 is part of the receiving device 268, the signalgenerator 800 communicates with other computing devices such as a socialnetworking system 228 or other servers to receive the haptic illusionsignal 202.

The signal generator 800 may include, among other components, aprocessor 804, a haptic interface circuit 806, a communication module808, memory 813 and a bus 810 connecting these components. The signalgenerator 800 may include other components not illustrated in FIG. 8Asuch as user interface modules for interacting with users or speakers.The signal generator 800 may also be a part of a larger device or anadd-on device that expands function of another device.

The processor 804 reads instructions from the memory 813 and executesthem to perform various operations. The processor 804 may be embodiedusing any suitable instruction set architecture, and may be configuredto execute instructions defined in that instruction set architecture.The processor 804 may be general-purpose or embedded processors usingany of a variety of instruction set architectures (ISAs), such as thex86, PowerPC, SPARC, RISC, ARM or MIPS ISAs, or any other suitable ISA.Although a single processor is illustrated in FIG. 8, the signalgenerator 800 may include multiple processors.

The haptic interface circuit 806 is a circuit that interfaces with thecutaneous actuators 802. The haptic interface circuit 806 generatesactuator signals 812 based on commands from the processor 804. For thispurpose, the haptic interface circuit 806 may include, for example, adigital-to-analog converter (DAC) for converting digital signals intoanalog signals. The haptic interface circuit 806 may also include anamplifier to amplify the analog signals for transmitting the actuatorsignals 812 over cables between the signal generator 800 and thecutaneous actuators 802. In some embodiments, the haptic interfacecircuit 806 communicates with the actuators 802 wirelessly. In suchembodiments, the haptic interface circuit 806 includes components formodulating wireless signals for transmitting to the actuator 802 overwireless channels.

The communication module 808 is hardware or combinations of hardware,firmware and software for communicating with other computing devices.The communication module 808 may, for example, enable the signalgenerator 800 to communicate with the social networking system 228, atransmitting device 240 or a speech source 318 over the network 260. Thecommunication module 808 may be embodied as a network card.

The memory 813 is a non-transitory computer readable storage medium forstoring software modules. Software modules stored in the memory 813 mayinclude, among others, applications 814 and a haptic signal processor816. The memory 813 may include other software modules not illustratedin FIG. 8, such as an operating system.

The applications 814 uses haptic output via the cutaneous actuators 802to perform various functions, such as communication, gaming, andentertainment. At least one of these applications 814 uses illusions ofmotions or actions within a body created by the operation of thecutaneous actuators 802, as described below in detail with reference toFIG. 8B through FIG. 8F.

The haptic signal processor 816 is a module that determines the actuatorsignals 812 to be generated by the haptic interface circuit 806. Thehaptic signal processor 816 generates digital versions of the actuatorsignals and sends to the haptic interface circuit 806 via bus 810. Adigital version of the actuator signals include information defining theanalog actuator signals to be generated by the haptic interface circuit806. For example, the digital version of the actuator signals mayindicate, for example, the amplitude or frequency of the analog actuatorsignals, time at which the actuator signals are to be transmitted by thehaptic interface circuit 806, and waveform of the actuator signals. Thehaptic signal processor 816 receives commands from the applications 814and determines parameters associated with the actuator signal 812. Theparameters of the actuator signal 812 may include, among others, timinggap between activation of the actuator signals, duration of the actuatorsignals, the amplitude of the actuator signals, the waveform of theactuator signals, which actuator signals to become active, and modes ofcutaneous actuators (if the cutaneous actuators have more than one modeof operation).

The haptic processor 816 may include sub-modules such as an inner-bodyillusion module 818, an interference signal processor 820 and afrequency decoder module 822. The inner-body illusion module 818 isinvoked to generate actuator signals 812 that cause the cutaneousactuators to generate the sensation or illusion of motions or actionsoccurring inside the body, as described below with reference to FIGS. 8Bthrough 8G. The interference signal processor 820 is responsible forgenerating actuator signals 812 that causes cutaneous actuators togenerate vibrations that result in constructive or destructiveinterference on the receiving user's skin, as described below in detailwith reference to FIGS. 9A through 9L. The frequency decoder module 822is responsible for generating actuator signals 812 in an operating modewhere the speech signal 216 is encoded using a frequency decompositionscheme, as describe below in detail with reference to FIGS. 10A through10E. The haptic processor 816 may include other modules for operatingthe cutaneous actuators to operate in different modes or performadditional or alternative functionality.

The signal generator 800 as illustrated in FIG. 8A is merelyillustrative and various modification may be made to the signalgenerator 800. For example, instead of embodying the signal generator800 as a software module, the signal generator 800 may be embodied as ahardware circuit.

FIG. 8B is a diagram illustrating cutaneous actuators 802A, 802B on apart of receiving user's body 824 (e.g., torso, forearm, leg, head) togenerate illusion of actions or motions occurring within the receivinguser's body, according to an embodiment. In the embodiment of FIG. 8B,two cutaneous actuators 802A, 802B are located at opposite sides of theuser's body part 824. The cutaneous actuators 802A, 802B can be any typeof actuators but vibrotactile actuators, such as voice coils and motors,are better adapted to providing these inner body illusions.

By activating the two cutaneous actuators 802A, 802B with apredetermined time interval, the user perceives motions or actionsoccurring within the user's body part 824 instead of the user's skin.For example, an illusory sensation such as (i) a bullet or projectileentering from a patch of skin and exiting through another path of skin,(ii) a virtual actuator vibrating inside the body part, (iii) a centerpoint of motion moving gradually from one point to another, (iv)saltation occurring inside the body part can be generated. Bycontrolling parameters such as the amplitude, timing and/or sequence ofactivation, the motion or action inside the user's body part 824 can beemulated.

More than two cutaneous actuators may be employed to create more diverseinner body sensations. FIG. 8C is a schematic diagram illustrating threecutaneous actuators 802C, 802D, 802E for generating sensory illusions ofa center point of vibrations in the receiving user's body 824 (e.g., theuser's head) moving from point 834A to 834B, according to an embodiment.A first cutaneous actuator 802C is mounted on the top, a secondcutaneous actuators 802D is attached to one side of the body part 824,and the last cutaneous actuators 802E is attached to the opposite sideof the body part 824. By activating the cutaneous actuators in a certainsequence, the sensation of actions or motions in the user's body part824 can be generated. For example, by activating cutaneous actuators802C simultaneously with cutaneous actuator 802D followed by additionvibrations from cutaneous actuator 802E causes the sensation ofvibrations to move from a center point 834A between the cutaneousactuators 802E, 802D to another location 834B closer to the cutaneousactuator 802C.

FIGS. 8D through 8G are timing diagrams illustrating the waveforms ofactuator signals 830A, 830B applied to two cutaneous actuators,according to embodiments. The actuator signals 830A, 830B are assumed tocause corresponding cutaneous actuators to generate vibrations withapproximately the same wave patterns. By modulating the amplitude and/ortime of the vibrations at the actuators 802A, 802B, various illusorysensation can be generated.

FIG. 8D illustrates waveforms for generating the sensation of a motionfrom a patch of skin with the cutaneous actuator 802A to another path ofskin with the cutaneous actuator 802B. By separating the on-set time ofthe two cutaneous actuators 802A, 802B by temporal distance of tp, asensation of an object moving from one cutaneous actuator to the othercutaneous actuators through the body can be created. In one embodiment,the temporal distance tp is set according to the following equation:

tp=a×d+b  (1)

where a and b are constants and d is the duration of the vibrations. Thefrequency of the vibrations may be 20 to 300 Hz, the amplitude of thevibration may be 10 to 35 dB and the duration of vibrations may bebetween 20 milliseconds and 5 seconds.

FIG. 8E illustrates waveforms for generating the sensation of a virtualactuator between the cutaneous actuators 802A, 802B, according to oneembodiment. This is a special case of FIG. 8D where tp is set to 0. Whenthe two cutaneous actuators 802A, 802B are activated simultaneously, asensation of vibrations occurring at a point between the two cutaneousactuators 802A, 802B are created. By controlling the amplitude of thevibrations, the point of virtual actuator may be moved closer to eitherof the two cutaneous actuators 802A, 802B.

Taking an example of creating the sensation of a virtual actuatorvibrating at an amplitude of A at location 834B of FIG. 8C, theamplitudes A_(C), A_(D), A_(E) of the vibrations generated by threeactuators 802C, 802D, 802E are set to A_(C)=α×A, A_(D)=β×A andA_(E)=γ×A, respectively, where α, β, and γ are determined as functionsof distances r_(C), r_(D), r_(E) from the cutaneous actuators to thevirtual actuator location. In one embodiment, α, β, and γ are determinedas follows:

α=r _(C)/(r _(C) +r _(D) +r _(E))  (2)

β=r _(D)/(r _(C) +r _(D) +r _(E))  (3)

γ=r _(E)/(r _(C) +r _(D) +r _(E))  (4)

In other embodiments, α, β, and γ are determined as functions ofBarycentric coordinates a_(C), a_(D), a_(E) of the location of thecutaneous actuators 802C, 802D, 802E using one of the followingequations:

A _(C) =a _(C) ×A; A _(D) =a _(D) ×A; A _(E) =a _(E) ×A  (5)

A _(C)=log(a _(C)+1)/log(Amax+1)×A; A _(D)=log(a _(D)+1)/log(Amax+1)×A;A _(E)=log(a _(E)+1)/log(Amax+1)×A  (6)

A _(C)=√{square root over (a _(C))}×A; A _(D)=√{square root over (a_(D))}×A; A _(E)=√{square root over (a _(E))}×A  (7)

where Amax is a_(C)+a_(D)+a_(E).

FIG. 8F illustrates waveforms for generating the sensation of panning ofvibrations from a point at or closer to the cutaneous actuator 802A to apoint at or closer to the cutaneous actuator 802B, according to oneembodiment. In this embodiment, the actuator signal 830A peaks at timeT3 and then gradually decreases until it reaches zero at time T4 (whichis temporally separated by duration d which is duration of the ramp,from time T3). The actuator signal 830B, on the other hand, starts toincrease from time T3 and gradually increases until time T4. Theincrease may be linear increase. When the actuator signals 830A and 830Bare applied to the cutaneous actuators 802A and 802B, the receiving userexperiences sensation vibrations moving from a patch of skin placed withthe cutaneous actuator 802A to another patch of skin placed with thecutaneous actuator 802B over time d.

FIG. 8G illustrates waveforms for generating the sensation of saltationfrom a point at or closer to the cutaneous actuator 802A to a point ator closer to the cutaneous actuator 802B, according to one embodiment.To generate such sensation, a number of pulses are sent in the actuatorsignal 830A to activate the cutaneous actuator 802A. Each pulse may havethe same amplitude of A, the same duration of d, and offset by atemporal distance of tp. Subsequently, a number of pulses are sent inthe actuator signal 830B to activate the cutaneous actuator 802B. Eachpulse of the actuator signal 830B has the same amplitude and the sameduration as those of the actuator signal 830A. To create such sensation,the duration of each pulse is shorter than 100 ms and satisfies thefollowing equation:

(tp−d)<d  (8)

That is, the off-times (tp−d) between the actuator signals 830A, 830Bare shorter than the duration d of the on-times of the actuator signals830A, 830B.

FIG. 8H is a flowchart illustrating a method of operating cutaneousactuators to produce illusion of actions or motions occurring within thereceiving user's body, according to one embodiment. The digital versionsof first activation signal and second activation signal is generated 840at inner-body illusion module 818. These digital signals may begenerated based on commands from one or more applications 814.

The digital version of the first activation signal is then converted 840into an analog signal for sending over to a first cutaneous actuatorover a wireless or wired communication by the haptic interface circuit806. Then, the converted analog first activation signal is sent 848 tothe first activation actuator. As a result, vibrations is generated bythe first cutaneous actuator.

Similarly, the digital version of the second activation signal isconverted 852 into an analog signal for sending over to a secondcutaneous actuator over a wireless or wired communication by the hapticinterface circuit 806. Then, the converted analog second activationsignal is sent 856 to the first activation actuator. As a result,vibrations is generated by the second cutaneous actuator. The first andsecond cutaneous actuators may be placed with a part of the user's bodyin-between and the amplitude or timing of vibrations of the cutaneousactuators are modulated so that the illusion of actions or motionswithin the user's body is produced.

The processes and their sequences illustrated in FIG. 8 are merelyillustrative and various modifications may be made. For example, theconversion to analog signals for second activation signal may occurbefore sending the analog first activation signal to the first cutaneousactuator. Alternatively, the conversion to analog signals for the firstand second activation signals may occur in parallel.

The sensation or illusion of motions or actions within a user's bodypart provided by embodiments described herein may enhance immersivenessof application such as video games. Further, such sensations or illusionmay be adopted to widen vocabulary and/or increase bandwidth of hapticcommunication. Such illusions may produce instructional and directionalcues for navigation purposes and object interactions. Inside the bodyillusion may have medical applications, and could be used to massage thehead to reduce headaches and reduce motion sickness and balancedisorders.

Haptic Communication System Using Interference of Haptic Outputs

Embodiments also relate to enhancing haptic communication by using twoor more cutaneous actuators to create constructive or destructiveinterference patterns on the receiving user's skin. The actuator signalsfor the two or more cutaneous actuators are shaped and generated so thatthe two or more cutaneous actuators cause vibrations on the receivinguser's patch of skin to increase or decrease at different locations. Inthis way, various enhancement to haptic communication including, but notlimited to, (i) improved spatial resolution of the vibrations, (ii)creation of virtual cutaneous actuators, and (iii) providing additionalencoding schemes for embedding information in the vibrations can beachieved.

When multiple main cutaneous actuators are used, vibrations from onemain cutaneous actuators may propagate to regions of the receivinguser's skin near where other main cutaneous actuators are located.Because such over-propagation of vibrations from one main cutaneousactuator may confuse the receiving user with respect to vibrationsgenerated by other main cutaneous actuators, the main cutaneousactuators spaced apart with sufficient distance to prevent theconfusion. Embodiments provide a mechanism to prevent or reduce theover-propagation of vibrations by generating counter-vibrations atauxiliary actuators where the auxiliary actuators vibrate with anopposite phase and lower amplitude.

FIG. 9A is a diagram illustrating using auxiliary cutaneous actuators904A, 904B to enhance localization of vibrations from a main cutaneousactuator 902, according to one embodiment. The main cutaneous actuator902 has two adjacent auxiliary cutaneous actuators 904A, 904B placed atthe left and the right sides of the main cutaneous actuator 902. Allthree actuators 902, 904A, 904B may be placed on a substrate 906 orother mounting mechanism onto a patch of the receiving user's skin.

The main cutaneous actuator 902 is an actuator that generates mainvibrations on the substrate 906 to provide haptic communication to thereceiving user via the vibrations it generates. For this purpose, themain cutaneous actuator 902 receives its actuator signal 812 from thehaptic interface circuit 806 as defined by the interference signalprocessor 820 (refer to FIG. 8A). Curve 908 show the amplitude of mainvibrations generated by the main cutaneous actuator propagating inx-direction (horizontal direction)

The auxiliary actuators 904A, 904B generates counter-vibrations tolocalize the vibrations of the main cutaneous actuator 902. Thecounter-vibrations has an opposite phase (i.e., 180 degrees out of phaseat the locations of the auxiliary actuator 904A and 904B) relative tothe vibrations generated by main cutaneous actuator 902. Curves 910A,910B illustrate the propagation of the counter-vibrations alongx-direction. The auxiliary actuators 904A, 904B may be smaller than themain cutaneous actuator 902.

When the vibrations of the main cutaneous actuator 902 and thecounter-vibrations of interact, destructive interference occurs betweenthe vibrations and the counter-vibrations. The combined vibrations as aresult of the interference, as shown by curve 914 in zone 2, hasdecreased range of vibrations and a peak amplitude relative to mainvibrations (shown as curve 908). Although the counter-vibrations asshown by curves 912A, 912B tend to propagate along a further distance inzones 1 and 3 due to constructive interference with the vibrations ofthe main cutaneous actuator 902, the amplitude of the counter-vibrationsmay at a lower than a sensory threshold of the receiving user.Therefore, the counter-vibrations from the auxiliary actuators 904A,904B localize the main vibrations of the main cutaneous actuator 902,and increase the spatial resolution of haptic communication. Moreover,other mechanisms such as providing damper or making the substrate 906thicker in areas beyond zone 2 may be used in conjunction to reduce thepropagation of the counter-vibrations.

Although the embodiment of FIG. 9A illustrates the auxiliary actuators904A, 904B at opposite sides along x-direction for the sake ofexplanation, in practice, more than two auxiliary actuators may surroundthe main cutaneous actuator 902. Moreover, auxiliary cutaneous actuatorsmay have the same size as the main cutaneous actuator, or auxiliarycutaneous actuators may be of different sizes.

FIG. 9B is a diagram illustrating cutaneous actuators 918A through 918Cto enhance localization of vibrations of multiple cutaneous actuators,according to one embodiment. The embodiment of FIG. 9B is the same asthe embodiment of FIG. 9A except that all three actuators are maincutaneous actuators that generate vibrations for sensing by thereceiving user. In the example of FIG. 9B, curves 920A, 920B, 920Crepresenting vibrations generated by the cutaneous actuators 918A, 918B,918C, respectively, have the same shape, representing that they all havethe same peak amplitude.

The operation principle of the cutaneous actuators 918A through 918C arethe same as the embodiment of FIG. 9A. The side cutaneous actuators918A, 918C generate vibrations that are opposite in phase (180 degreesout of phase) relative to the vibrations generated by the centercutaneous actuators 918B. However, instead of relying oncounter-vibrations with smaller amplitude, the embodiment of FIG. 9Brelies on the destructive interference between vibrations of theadjacent cutaneous actuators to increase spatial resolution associatedwith the center cutaneous actuator 918B.

As shown in FIG. 9B, nodes N1 and N2 where the amplitude of the combinedvibrations become zero due to destructive interference of vibrationsoccurs within zone D. That is, the vibrations from the center cutaneousactuators 918B do not propagate beyond zone D, increasing the spatialresolution of the spatial communication.

The embodiment of FIG. 9B is advantageous, among other reasons, becauseno separate auxiliary cutaneous actuators is used and there is no needto fine tune the operational parameters (e.g., amplitude and phase ofcounter-vibrations) of the auxiliary cutaneous actuators.

Although the embodiment of FIG. 9B is explained using three cutaneousactuators aligned in horizontal direction for the sake of explanation,the same principle can be expanded to two dimensional pattern. FIG. 9Cis a diagram illustrating a lattice structure of cutaneous actuators923A through 923I, according to one embodiment. In FIG. 9C, cutaneousactuators 923E, 923F are surrounded by 6 other cutaneous actuators. Whentwo adjacent cutaneous actuators are activated, they may producevibrations of an opposite phase to localize their vibrations.

Constructive interference of vibrations generated at multiple cutaneousactuators may be used to create a virtual actuator. FIG. 9D is a diagramillustrating vibrations perceived by a receiving user when a cutaneousactuator 924A is activated, according to one embodiment. In FIG. 9D, theamplitude of the vibrations generated by the cutaneous actuator 924A isabove a threshold that can be sensed by the receiving user. The userwill sense the vibrations in region R1 around the cutaneous actuator924A. The region R1 will be wider or narrower depending on the amplitudeof the vibrations. When other cutaneous actuators 924B, 924C, 924D areindividually activated to generate vibrations above the sensorythreshold, the receiving user will sense vibrations in respectiveregions around the actuators.

When a cutaneous actuator (e.g., 924A) is weakly activated so that thegenerated vibrations do not exceed the sensory threshold, the receivinguser would not sense the vibration. However, when multiple cutaneousactuators are operated in conjunction to generate vibrations at the samephase, the constructive interference between the vibrations will createregions where the amplitude of the vibrations is above the sensorythreshold. For example, FIG. 9E is a diagram illustrating vibrationsdetected by the receiving user at region R2 when four cutaneousactuators 924A through 924D are activated to generate vibrations belowthe sensory threshold, according to one embodiment. The vibrationsgenerated by the cutaneous actuators 924A through 924D are in the samephase, and hence, the constructive interference of the vibrations resultin the largest amplitude in the center of R2 region, as if a cutaneousactuator is operated at the center of region R2.

By selectively operating the cutaneous actuators 924A through 924Dand/or controlling the amplitude of the generated vibrations, theperceived center location of the vibrations can be moved around. In thisway, virtual cutaneous actuators can be created to give the perceptionof providing more cutaneous actuators than the number of physicalcutaneous actuators.

By providing the cutaneous actuators with actuator signals thatgradually change over time, a virtual motion can be created. By usingthe same principle as the virtual cutaneous actuators but changing theamplitude of the actuator signals, the perceived center locations of thevibrations can be moved over time. This creates the sensation of thevirtual cutaneous actuators physically moving over a path.

FIG. 9F is a diagram illustrating waveforms of actuator signals 812Athrough 812D to cause the sensation of a virtual motion, according toone embodiment. The actuator signals 812A, 812B, 812C, 812D are fedcutaneous actuators 924A, 924B, 924C, 924D, respectively. The actuatorsignals 812A and 812C have the same pattern (although the signals 812Aand 812C are 180 degrees out of phase with each other) whereas theactuator signals 812C and 812D have the same pattern (although thesignals 812C and 812D are 180 degrees out of phase with each other).

FIG. 9G are diagrams illustrating a virtual motion created by thewaveforms of FIG. 9F, according to one embodiment. As the actuatorsignals 812A through 812D are fed to the cutaneous actuators 924Athrough 924D, two center points of vibrations are created. Thevibrations are 180 degrees of phase with each other, and are separatedby a region that has zero amplitude. The center points rotate in aclockwise direction, creating the sensation of the virtual movement.

In the embodiment of FIG. 9G, the centers of the vibrations are locatedbeyond the four cutaneous actuators 924A through 924D. Sensations can becreated beyond the dimension of the array of the cutaneous actuatorarray. This is advantageous, among other reasons, because a compacthaptic device can be used to provide vibrational sensation over a largerarea.

By applying different waveforms and different configuration of thecutaneous actuator array, various virtual motion can be embodied.

FIG. 9H is a diagram illustrating placement of two cutaneous actuators904A, 904B applied with different frequencies, according to oneembodiment. By differing the frequencies applied to the cutaneousactuators, beat patterns of different frequencies can be generated byinterference between the vibrations generated by the cutaneousactuators. Such beat patterns may be used to encode information forhaptic communication.

FIG. 9I is a diagram illustrating a vibration pattern when the samefrequency of vibrations are generated by the two cutaneous actuators904A, 904B, according to one embodiment. The vertical axis of FIG. 9Grepresents distance D from a center between the two cutaneous actuators904A, 904B, and the horizontal axis represents time. When the twoactuators produce vibrations of the same frequency, no apparent beatpattern is observed as shown in FIG. 9I.

FIGS. 9J and 9K are diagrams illustrating vibration patterns when thetwo cutaneous actuators 904A, 904B generate vibrations of differentfrequencies, according to one embodiment. Specifically, FIG. 9Jillustrates the vibration pattern when the cutaneous actuator 904Agenerates vibrations with frequency (f1) of 249 Hz and the cutaneousactuator 904B generates vibrations with frequency (f2) of 250 Hz. Asshown in FIG. 9J, the vibrations pattern generated as a result include abeat pattern of a lower frequency. FIG. 9K illustrates the vibrationpattern generated when the cutaneous actuator 904A generates vibrationswith frequency (f1) of 245 Hz and the cutaneous actuator 904B generatesvibrations with frequency (f2) of 250 Hz. As the difference increases,the resulting vibrations exhibit a beat pattern of a higher frequency.

Hence, by leveraging the varying beat pattern resulting from thedifference in the frequency difference between the vibrations of thecutaneous actuators 904A, 904B, information for haptic communication canbe embedded in the vibrations.

FIG. 9L is flowchart illustrating using interference of vibrations fromat least two cutaneous actuators for enhanced haptic communication,according to one embodiment. After receiving instructions on hapticinformation to convey to the receiving user, the interference signalprocessor 820 determines 940 the phase and amplitude of multipleactuator signals. For this purpose, the interference signal processor820 may reference the receiving user's sensory threshold, configurationof the cutaneous actuator array, and receiving user's preferences orsettings.

Based on the determination of the interference signal processor 820, thehaptic interface circuit 806 generates 944 the actuator signals. Suchprocess may include performing digital to analog conversion based oninstructions from the interference signal processor 820.

The generated actuator signals are then sent 948 to correspondingcutaneous actuators to cause constructive or destructive interferencesof vibrations on the receiving user's patch of skin.

The process illustrated in FIG. 9L is merely illustrative and variousmodifications may be made to the process. For example, generating 944the actuator signals may be performed in parallel with sending 948 ofthe generated actuator signals.

Selectively Using Dominant Frequency Bands in Haptic CommunicationSystem

Embodiments also relate to performing haptic communication usingfrequency decomposition of a speech where dominant frequencies of thespeech is detected and then sent to a signal generator to actuateactuators mapped to the dominant frequencies. The digitized version ofthe speech is segmented into a plurality of frames and then apredetermined number of dominant frequencies are detected from eachframe. The dominant frequencies are sent over to the signal generator sothat the actuators corresponding to the dominant frequencies areactivated for a time period. The activation time period may correspondto the frame or be shorter or longer than the frame.

The acoustic structure of speech is characterized by multiple peaks inthe frequency spectrum (i.e., formants) generated by resonances in thevocal tract. These formants are the one of the primaryinformation-carrying features of speech for many phonemes, especiallyvowels. In frequency decomposition algorithms, adjacent frequency bandsare mapped to spatially adjacent cutaneous actuators on the skin. Inlight of the poor spatial resolution and poor intensity discriminabilityof the skin, it is challenging for a receiving user to identify thelocation of the cutaneous actuators with the greatest intensity,especially when adjacent actuators are also activated due to thetypically broad shoulders of the formant peaks in the spectrum.Embodiments identify frequencies with local maxima and the correspondingcutaneous actuators are activated while other cutaneous actuators arenot activated. In this way a clearer unique haptic signature ispresented for each unique formant pattern by the activation of thecutaneous actuators. This approach can be used alone or in a hybridsystem in which vowels (i.e., prominent formant structure) and/ornon-vowels are encoded.

FIG. 10A is a block diagram of the speech source 318 performingfrequency decomposition using selective dominant frequencies, accordingto one embodiment. The speech source 318 generates the speech signal 216by processing voice signals (e.g., received from a source such as amicrophone 1002) using frequency decomposition. As described above withreference to FIGS. 3 and 8, the speech signal 216 may be sent directlyover the network 260 or via the social networking system 228 to areceiving device 268 (including, e.g., the signal generator 800) fordecoding and generating actuator signals 812. The speech source 318 mayinclude, among other components, the microphone 1002, a noise filter1004, an analog-to-digital (AD) converter 1008, and a frequencydecomposition (FD) encoder 1010.

The microphone 1002 is a hardware component for capturing a sendinguser's voice. Instead of microphone 1002, the speech source may includeother voice sources such as a storage medium (e.g., CD or DVD player) ora speech synthesizer that generates voice signal from text. Themicrophone 1002, in response to detecting the sending user's voice,generates a voice signal 1030 in an analog format.

The noise filter 1004 receives the voice signal 1030 and performsfiltering using circuit components to generate a filtered voice signal1032. The noise filter 1004 may use various well know technique ortechniques to be developed in the future to reduce noise and amplifyfrequency components of the voice signal 1030 corresponding to thesending user's voice.

The AD converter 1008 receives the filtered voice signal 1032 anddigitizes it to generate a digital version 1034 of the voice signal forfurther processing by the FD encoder 1010. For this purpose, the ADconverter 1008 includes components for sampling the filtered voicesignal 1032 and quantizing the sampled voice signal.

The FD encoder 1010 performs frequency decomposition of the digitizedvoice signal 1034 using dominant frequencies, and generates the speechsignal 216 as a result. The FD encoder 1010 may include, among othercomponents, a processor 1012, a signal interface 1022, a networkcommunication module 1024, a memory 1013 and a bus 1011 connecting thesecomponents. Although the FD encoder 1010 is illustrated in FIG. 10A asbeing a microprocessor capable of various operations, the FD encoder1010 may be embodied as a dedicated hardware circuit for performing thefrequency decomposition.

The processor 1012 reads and executes instructions, for example, storedin the memory 1013. Although a single processor is illustrated in FIG.10A, a plurality of processors may be used in parallel. In someembodiments, the processor 1012 is a circuitry specifically designed fordigital signal processing.

The signal interface 1022 is an interface circuit for receiving thedigitized voice signal 1034 from the AD converter 1008. The signalinterface 1022 may use well known serial or parallel bus schemes toreceive the digitized voice signal 1034 and/or other signals.

The network communication module 1024 includes hardware circuit thatformats and sends the speech signals 216 to the signal generator 800.The speech signals 216 include frame information and dominant frequencyinformation. The network communication module 1024 may perform, amongother operations, packetizing the frame information and the dominantfrequency information for transmittal to the signal generator 800.

The memory 1013 is a non-transitory computer readable storage mediumthat stores software modules. The memory 1013 may store, among othermodules, a frame splitter 1016, a Fast Fourier Transformation (FFT)module 1018 and a peak selector 1119. The memory 1013 may include othersoftware components illustrated in FIG. 10A, such as an operation system(OS).

The frame splitter 1016 splits the digitized voice signal 1034 intoframes. FIG. 10B is a diagram illustrating splitting the digitized voicesignal 1034 into multiple frames FR1 through FRN, according to oneembodiment. For example, when 44.1 kHz sampling rate is used for thedigitized voice signal 1034, each frame may include 1024 samplescorresponding to 23.2 ms. The frame length may be long enough so thatadequate frequency resolution is maintained, but short enough so thesignal can be viewed as statistically stationary over the duration ofthe frame (i.e., short enough so that the statistics of the signal areapproximately constant). In addition, for digital signal processing, itis convenient to use a frame length whose number of samples is a powerof 2. As shown in FIG. 10B, frames can overlap (for example, 50% overlapbetween adjacent frames).

The FFT module 1018 performs Fast Fourier transform on a portion of thedigitized voice signal of each frame in a time domain to a frequencydomain. As a result of FFT, an approximate power spectrum of the portionof the digitized voice signal over the corresponding frame is obtained.The signal power is clustered in a relatively small number of locationsin the frequency spectrum, which correspond to the speech formants. Thefine fluctuations are due in part to numerical artifacts but these donot affect the dominant frequencies and hence, no further technique isapplied to reduce the artifacts. FIG. 10C is a graph illustrating theresult of the Fast Fourier transformation, according to one embodiment.In other embodiments, spectral estimation techniques other than FFT maybe used to transform the portion of digitized voice signal in a timedomain to a frequency domain.

In one embodiment, the peak selector 1019 sets portions of the frequencydomain spectrum that is outside the frequency range of interest (e.g.,200 Hz-8.4 kHz) to zero. Then, a frequency location of the highest peakmagnitude (i.e., a first dominant frequency) in the frequency domainspectrum is identified. After the location is identified, a windowsurrounding the peak by +/−400 Hz is set to zero, so that any relativepeak magnitude within 400 Hz of the identified peak is ignored. The nexthighest peak (i.e., a second dominant frequency) is identified, and awindow surrounding the next highest peak magnitude by +/−400 Hz is setto zero. The same process is repeated to identify a predetermined numberof peaks or dominant frequencies. FIG. 10D illustrates five peaks N1through N5 of the dominant frequencies, according to one embodiment. Theframe number and the corresponding dominant frequencies and amplitudesin the frame are sent to the signal generator 800 as the speech signal216 by the network communication module 1024.

The frequency domain spectrum is divided into a number of channels(e.g., 32 channels) between the lower and upper cutoff frequencies(e.g., 200 Hz and 8.4 kHz, respectively). In one embodiment, thechannels are linearly spaced below 1 kHz and spaced logarithmicallyabove 1 kHz. Each peak identified above is associated with the channelwith the closest center frequency.

After the signal generator 800 receives the speech signal 216, thefrequency decoder module 822 (refer to FIG. 8A) extracts the framenumber and the corresponding dominant frequencies. The frequency decodermodule 822 may perform pre-processing to enhance the hapticcommunication. The pre-processing may include, for example, adjustingthe playback time of the speech signal 216. In one embodiment, theduration of the frame is preserved at the frequency decoder module 822so that the frames at the speech source 318 and frames at the signalgenerator 800 are of the same duration. The frames at the signalgenerator 800 can be played successively, so a nonzero overlap on inputframes results in a slower than real-time playback on output.Alternatively, the degree of overlap can be preserved on output, or adifferent, arbitrary, amount of overlap can be used. Alternatively, theoutput frame length can be made different from the input frame length.In addition, averaging of successive output frames playback can beimplemented. In general, the playback rate can be slower or faster thanthe original recording rate.

The frequency decoder module 822 also determines the cutaneous actuators802 to be activated. Each cutaneous actuator may be mapped to a certainchannel and an actuator may be activated for a time representing a framewhen the speech signal 216 indicates that a dominant frequencycorresponding the actuator is present in the frame. In one embodiment,the cutaneous actuators 802 are driven with an intensity that isproportional to the magnitude of respective peaks of the dominantfrequencies. In another embodiment, the cutaneous actuators 802 aredriven with a fixed intensity, independent of the magnitude of thecorresponding spectral peaks of the dominant frequencies.

Although above embodiments describe the speech source 318 and the signalgenerator 800 as being located remotely from each other, the speechsource 318 may be co-located with the signal generator 800 andcommunicate via wire or short-distance communication. Further, thespeech source 318 and the signal generator 800 may be embodied as asingle device.

FIG. 10E is a diagram illustrating placing of cutaneous actuators on thereceiving user 1050 for reproducing the voice signal using frequencydecomposition, according to one embodiment. The signal generator 800outputs actuator signals 812A through 812N to the cutaneous actuators802A through 802N where each cutaneous actuator corresponds to a channelof dominant frequency as described above with reference to FIG. 10D.

In the embodiment of FIG. 10E, the first channel has a center frequencyof 387.6 Hz and is mapped to a cutaneous actuator that is close to theleft wrist. The second channel has a center frequency of 689.1 Hz and ismapped to a cutaneous actuator that is about an inch away from thefirst, placed more proximally on the left arm. Channel 16 is mapped to acutaneous actuator near the left shoulder. Channel 17 is mapped to acutaneous actuator that is near the right wrist, and progressivelyhigher channels appear more proximally at higher locations with channel32 near the right shoulder.

Further, each of the channels in the embodiment of FIG. 10E has thefollowing center frequencies in kHz: 0.3876, 0.6891, 0.9905, 1.2920,1.4212, 1.5073, 1.6365, 1.7657, 1.8949, 2.0241, 2.1964, 2.3256, 2.4979,2.6701, 2.8424, 3.0577, 3.2730, 3.4884, 3.7037, 3.9190, 4.1774, 4.4358,4.7373, 4.9957, 5.3402, 5.6417, 5.9862, 6.3308, 6.6184, 7.1490, 7.2366and 8.0104. Each dominant frequency in the speech signal 216 is mappedto the channel with the closest center frequency.

FIG. 10F is a flowchart illustrating a process of performing hapticcommunication using selective dominant frequencies, according to oneembodiment. The speech source 318 receives the voice signal from asource, and then generates 1062 a digitized speech signal. The digitizedspeech signal is segmented 1064 into a plurality of frames.

The segmented frames are then processed by performing 1066 Fast FourierTransform on each frame. Then, the speech source 318 detects 1068 apredetermined number of dominant frequencies with highest peakamplitudes in the result of the FFT. Spectral estimation techniquesother than FFT may be applied to the speech source 318 to generate afrequency domain spectrum. The speech source 318 sends 1070 dominantfrequency information and the frame information as the speech signals tothe signal generator 800.

The signal generator 800 receives 1072 the dominant frequencyinformation and the frame information from the speech source 318. Thenthe signal generator 800 pre-processes 1074 the dominant frequencyinformation and the frame information by adjusting the length, sequenceor overlapping of the frames.

The signal generator 800 also generates 1076 actuator signal based onthe mapping of dominant frequencies to the actuator. Then, the generatedactuator signals are transmitted 1078 from the signal generator 800 tothe cutaneous actuators.

The sequence and processes as described in FIG. 10F is merelyillustrative and various modifications can be made. For example, insteadof performing segmentation 1064 on the digitized speech, analog signalmay be segmented and then digitization may be performed thereafter.Further, pre-processing 1074 may be performed in parallel with or at thesame time of generating 1076 of the actuator signals.

Haptic Communication System Using Haptic Symbol Set

Embodiments also relate to a haptic symbol set which specify sequencesof haptic signals to be generated from input words of a language, suchas English. As noted above, various methods can be used to convert aninput sequence of words into an output sequence of haptic signals. Inone embodiment, the haptic signals are based on the phonemes of theinput sequence of words.

FIG. 11A illustrates an exemplary haptic device 1102 with cutaneousactuators that can be used to transmit sequences of haptic illusionsignals to a receiving user based on a sequence of input words,according to one embodiment. The haptic device 1102 includes a harness1103, upon which are attached a two dimensional array of cutaneousactuators 1104 which, as described above, contact the skin of the user'sforearm 1101 and are able to exert a vibrations upon the user's forearm.Each cutaneous actuator 1104 may also include a rubber tip 1105.

The harness 1103 affixes the cutaneous actuators 1104 to fixed positionsfacing the skin on the dorsal side of the forearm 1101. The harness 1103may wrap around the receiving user's forearm 1202 when fitted on thereceiving user's forearm 1202. The harness 1103 may be made of anycombination of elastic or inelastic fabric, such as rayon, nylon, foams,rubbers, polymers, cotton, etc. The harness 1103 may further includerigid members, made of plastic, metal, or other rigid materials in orderto provide structural support for the harness 1103, to protect sensitivecomponents embedded in the harness 1103, or to cause the harness 1103 tobe affixed at a certain position on the user's forearm 1101. In oneembodiment, the harness 1103 includes a fitted rigid member that iscontoured according to the shape of a wrist of a human of average heightand build and of a particular age (and gender). The harness 1103 isbuilt around this fitted rigid member such that a wearer of the harness1103 wears the harness 1103 in a certain position which conforms thefitted rigid member to the contour of the shape of the receiving user'swrist. If the harness 1103 is worn in any other position, the fittedrigid member may cause discomfort for the receiving user as it does notmatch the contour of the user's arm in that position. Thus, the harness1103 can be designed to be worn in only a limited number of positions,which in turn causes the cutaneous actuators 1104 to be in a similarposition (within a threshold range of distances) each time the harness1103 is worn by the receiving user. Instead of rigid members, in otherembodiments, non-rigid members, such as thicker fabrics (e.g., exceeding1 cm), fabric cutouts, and so on, may be used to achieve the sameeffect. As an example, the harness 1103 may be configured such that thetwo dimensional array of cutaneous actuators 1104 are placed 2 cm in aproximal direction from user's wrist, and such that the two dimensionalarray of cutaneous actuators 1104 are situated medially on the dorsalside of the user's forearm 1101. Additionally, instead of the user'swrist, any another anatomical feature of the user's forearm (or user'sbody) may be used instead as a landmark to affix the haptic device.

The harness 1103 may further be adjustable such that its dimensions maybe changed to become form fitting with the user's forearm 1101. This maybe achieved with straps which connect a portion of the harness 1103 withanother portion, and which may be adjustable in length or includeremovably attachable material, such as Velcro, on one surface of thestrap, in order to attach the strap to the harness 1103 along differentpoints of the strap. This causes the dimensions of the harness, such asits diameter, to change, and allows the harness 1103 to be form fittingto the user's forearm 1101. Alternatively, or in combination, theharness 1103 may include elastic materials, which cause the harness tocompress against the user's forearm 1101, but which also allow expansionwhen the user initially puts the harness 1103 on his or her forearm1101.

Attached to the harness 1103 may be a signal generator (not shown), suchas signal generator 800, in order to generate actuator signals 272,which are transmitted to the cutaneous actuators 1104. In oneembodiment, the signal generator generates sequences of actuator signalswhich are representative of phonemes of input words. In this fashion, auser, after training, is able to understand phonemes via haptic feedbackrather than audio or visual transmission of the words (e.g., via speechor written words). This can be significantly advantageous for users whomay have difficulties communicating using audio or visual means, and mayalso be useful where such audio or visual communication are unavailable,such as in a loud environment with low available light (e.g., a searchand rescue operation). Additional details regarding these sequences ofactuator signals are described below with reference to FIGS. 11B through11J.

The cutaneous actuators 1104 generate a haptic output (e.g., vibrations)which can be sensed by the skin of the user's forearm. In oneembodiment, the cutaneous actuators 1104 are placed on an interiorsurface of the harness 1103 facing the skin of the user's forearm. Asused here, cutaneous actuator(s) 1104 may refer to a single cutaneousactuator 1104 or the two dimensional array of cutaneous actuators 1104which are attached to the harness 1103 as shown. The haptic outputgenerated by each of the cutaneous actuators 1104 may include a force orpressure or other output that may be sensed by the user's skin, such aselectrical stimulus, temperature stimulus (e.g., heat), negativepressure (e.g., suction), vibrational output, shear movement stimulus,and so on. The cutaneous actuators 1104 may be similar to the cutaneousactuators described above, such as cutaneous actuators 108, 208, and802. The cutaneous actuators 1104 are arranged in a two dimensionalarray on the harness 1103, and with their orientation such that anactivation of a cutaneous actuator 1104, which generates the hapticoutput, causes a sensation to be sensed by the user. In the case wherethe cutaneous actuators 1104 provide a force sensation (e.g., pressure),the cutaneous actuators 1104 are arranged such that the force sensationgenerated by the cutaneous actuators 1104 is directed towards the user'sskin.

The cutaneous actuators 1104 receive actuator signals 1111 from a signalgenerator or other device, and in response to the receipt of theactuator signals 1111, generates a haptic output. The haptic output maybe generated within a threshold response time from the receipt of theactuator signal (e.g., 10 ms). When the cutaneous actuator 1104generates a haptic output, it is activated, transitioning from anon-activated state. Furthermore, the cutaneous actuators 1104 may havea ramp up and ramp down interval that is within a threshold interval(e.g., 1 ms). This is the time required for the cutaneous actuators 1104to go from a state of non-activation to a state of full activation. Asnoted above, the cutaneous actuators 1104 may be of many types,including voice coils, linear resonant actuators (LRAs), eccentricrotating mass actuators, piezoelectric actuators, electroactive polymeractuators, shape memory alloy actuators, pneumatic actuators,microfluidic actuators, and acoustic metasurface actuators, and so on.The actuator signal generated by the signal generator is such that thesignal will activate the cutaneous actuators 1104 based on the type ofthe cutaneous actuators 1104, and may include both a signal as well asthe power needed to drive the cutaneous actuator 1104. By providing asequence of haptic outputs to the user, the haptic device 1102 is ableto generate a pattern which can be recognized (i.e., sensed) by thereceiving user and which allows for the user to discern information fromthe pattern. Additional details regarding generating this sequence ofhaptic outputs is described below with reference to FIGS. 11B through11J.

Although a two by six array of cutaneous actuators 1104 is shown inFIGS. 11C, 11D and 11H, this configuration is simply for the purposes ofillustration and in other embodiments the array of cutaneous actuators1104 may include fewer or more cutaneous actuators 1104 and may bearrayed in a different fashion (e.g., three by six). The cutaneousactuators 1104 may be arranged in a non-regular pattern that differsfrom the regular pattern shown in FIGS. 11C, 11D, and 11H. Instead, thearrangement of the cutaneous actuators 1104 may be arbitrary.

In one embodiment, the cutaneous actuators 1104 each include a singlerubber tip 1105 which may contact the skin. Each rubber tip 1105 maycontact the user's skin when the respective cutaneous actuator 1104generates a haptic output. This may occur because the cutaneous actuator1104 pushes the rubber tip 1105 towards the user's skin. When the rubbertip 1105 contacts the user's skin, this causes the user to feel asensation (e.g., a touch sensation). The rubber tip 1105 may reduce thecontact area of the cutaneous actuators 1104 with the skin and focus thehaptic output provided by the cutaneous actuators 1104. They may have adiameter of 1 cm.

FIG. 11B is a block diagram illustrating the components of a system forconverting input words 1106 to actuator signals 1111 to activate thecutaneous actuators 1104, according to an embodiment. These componentsmay be stored as computer readable instructions in memory, such as thememory 813. In another embodiment, these components are separatelyimplemented using dedicated hardware, including a processor and memory.

The input words 1106 are words in any spoken language, such as English.Thus, as the input words 1106 can be spoken, the input words 1106 may beconverted into phonemes, which may, for example, be represented by theInternational Phonetics Alphabet (IPA). For example, the word “language”may be translated to the sequence of phonemes “‘læ

gwId

|.”

The word-haptic signal converter 1107 is a deterministic finite statemachine which accepts the input words 1106 and outputs a unique sequenceof output actuator signals 1111 for every sequence of non-homonymicinput words 1106. If two sequence of input words 1106 are homonyms ofeach other, then the word-haptic signal converter 1107 would generatethe same sequence of actuator signals 1111 for both sequences. Theword-haptic signal converter 1107 includes a word-phoneme converter1108, a phoneme-haptic signal converter 1109 and a phoneme-haptic signaldatabase 1110. In one embodiment, the word-phoneme converter 1108 may bea component in a signal generator, such as signal generator 800.

The word-phoneme converter 1108 converts the input words into sequencesof phonemes. The output of the word-phoneme converter 1108 may be in theform of a standard phonetic alphabet, such as the IPA, or may be inanother format, such as a compressed phonetic alphabet compressed tohave a number of phonetic symbols representing phonemes in thecorresponding language of the input words 1106 that corresponds to thetotal number of phonemes available in that language. The word-phonemeconverter 1108 may convert words to phonemes based on a word-phonemesequence database, or using a set of rules to convert letters to sounds.These rules consider the context of letters within an input word 1106 togenerate an approximate phonetic spelling of the word. In some cases, aninput word 1106 may be composed of multiple individual words, and whilethe input word 1106 has no correspondence in the word-phoneme sequencedatabase, the individual subparts do. The word-phoneme converter 1108may search an input word 1106 which is not recognized in theword-phoneme sequence database for sub-parts that are recognized, and ifrecognized, the word-phoneme converter 1108 may convert the individualsub-parts to their phonetic counterparts.

The phoneme-haptic signal converter 1109 converts the phonemes receivedfrom the word-phoneme converter 1108 to actuator signals 1111 using thephoneme-haptic signal database 1110. For each phoneme received from theword-phoneme converter 1108, the phoneme-haptic signal converter 1109accesses the phoneme-haptic signal database 1110 to determine thecorresponding sequence of actuator signals 1111 to output. In oneembodiment, each phoneme has a single corresponding sequence of actuatorsignals 1111 stored in the phoneme-haptic signal database. However, inother embodiments, the actuator signals 1111 corresponding to a phonemedepend upon other phonemes surrounding a phoneme in a word. In such acase, the phoneme-haptic signal converter 1109 may search for a phonemein the phoneme-haptic signal database 1110, but also search for thephoneme with its immediately preceding phoneme, its immediatelyfollowing phoneme, and a sequence including the immediately precedingphoneme, the phoneme itself, and the immediately following phoneme. Ifthe phoneme-haptic signal converter 1109 identifies multiple matches inthe phoneme-haptic-signal database 1110, the phoneme-haptic signalconverter 1109 may select one based on priority rules for that phoneme,or may use default rules, such as selecting first the match that matchesall three phonemes, then the match that matches the phoneme and itspreceding phoneme, and then the match that matches the phoneme and thefollowing phoneme, and so on. For example, the word apple has thephonetic translation of “‘æp

l,” which comprises the phonemes “ae,” “p,” “ah,” and “l.” In this case,a match in the phoneme-haptic signal database 1110 may indicate that thephoneme “p” preceded by “ae” should result in a different correspondingsequence of actuator signals 1111 than the phoneme “p” in any otherscenario. In that case, the different actuator signal is selected by thephoneme-haptic signal converter 1109 for output in correspondence withdetecting the “p” phoneme.

Note that the sequence of actuator signals 1111 that are generated arenot just patterned based on which cutaneous actuators 1104 to activatein sequence. Instead, the sequence of actuator signals 1111 may also bepatterned such that each actuator signal 1111 in the sequence may have adifferent duration, vibrational frequency, and so on. In addition, theabsence of a actuator signal 1111, combined with other actuator signals1111, may also indicate a particular sequence that corresponds to aphoneme. Thus, the duration of signals, frequency, and the lack ofsignals, may all contribute to different sequences of actuator signals1111 that represent different phonemes. Additionally, the actuatorsignals 1111 may not simply be binary (e.g., on and off) but may bevariable instead, allowing the haptic output of the cutaneous actuators1104 to vary in intensity, and allowing haptic outputs that followcomplex patterns and waveforms.

The actuator signals 1111 output by the word-haptic signal converter1107 are transmitted to the cutaneous actuators 1104. Each actuatorsignal 1111 activates one of the cutaneous actuators 1104 for a durationof time (and in some embodiments, for a particular intensity, frequency,and so on). At the point when the cutaneous actuator 1104 receives itsactuator signal 1104, it activates. When the same actuator signal 1111ceases, the cutaneous actuator 1104 deactivates. Alternatively, theactuator signals 1111 are digital in that they transmit a specifiedduration and start time (and the other factors described above, such asfrequency) for a specified cutaneous actuator 1104. This digital signalis converted by a processor or by the cutaneous actuators 1104themselves, and subsequently the corresponding cutaneous actuator 1104activates according to the specifications in the digital signal. Asequence of these actuator signals 1111 can generate a unique patternthat can be recognized and distinguished by a user from other sequencesof actuator signals 1111. In this fashion, the phonemes of the inputwords 1106 may be converted into haptic sensations via the word-hapticsignal converter 1107.

Note that the word-haptic signal converter 1107 may also be able togenerate sequences of actuator signals 1111 for non-word input, such aspunctuation, numbers, symbols, and other text elements which may be usedin the language. The sequences of actuator signals for these elementsmay also be stored in the phoneme-haptic signal database 1110.

A set of examples of particular actuator signals 1111 and a descriptionof an example correspondence between phonemes and actuator signals aredescribed below with reference to FIGS. 11C through 11I.

FIG. 11C illustrates an example of a haptic output for a singlecutaneous actuator from a single actuator signal, according to anembodiment. In the illustrated example, the haptic output is generatedby a single cutaneous actuator 1112 following the input of a singleactuator signal. A single actuator signal is the smallest unit of aactuator signal 1111 that, by itself, is able to cause a change in thecutaneous actuators 1104 when transmitted to the cutaneous actuators1104. In one embodiment, the haptic output caused by the single actuatorsignal is represented by the graph in FIG. 11C, which plots an intensityof the haptic output, whether it be force, vibration, pressure, or anyof the other types of sensations described above, versus time 1106. Inthis embodiment, the single haptic output has an attack 1113, duration1114, and a decay 1115. The attack 1113 occurs where the intensity ofthe haptic output “ramps up” or increases. In the case where the hapticoutput is a force-based output by the single cutaneous actuator 1112,the attack 1113 indicates that the force does not increase to a maximumlevel instantaneously, but that the increase to the maximum intensitylevel occurs over a period of time. After the maximum intensity level(as specified by the actuator signal 1111) is reached, the maximumintensity level is maintained for a duration 1114. This duration mayalso be specified by the actuator signal 1111. Subsequently, theactuator signal 1111 indicates that the intensity should return to abaseline, such as zero. Thus, the haptic output enters a decay 1115,where a time interval exists between the intensity of the haptic outputat the maximum level and the intensity of the haptic output reaching thebaseline value. In one embodiment, this single haptic output representsa consonant phoneme of an input word 1106. Thus, if a consonant phonemeis encountered in an input word, the corresponding actuator signal 1111which is outputted causes a corresponding single cutaneous actuator togenerate a haptic output that may be similar to the one shown in FIG.11C, i.e., a single activation of a specified duration and to a maximumintensity level.

FIG. 11D illustrates an example of haptic outputs for two cutaneousactuators from a sequence of two actuator signals, according to anembodiment. While FIG. 11C illustrated a single actuator signal thatcaused a single cutaneous actuator to generate a haptic output, FIG. 11Dillustrates a sequence of two actuator signals that cause each of thetwo different cutaneous actuators to generate a haptic output. Eachactuator signal is transmitted to a different cutaneous actuator 1104.In the illustrated example, one of the actuator signals is transmittedto the first cutaneous actuator 1117, and another of the actuatorsignals is transmitted to the second cutaneous actuator 1118 (with theselected cutaneous actuator indicated by a bolded outline). The twoactuator signals are modulated such that the haptic output for the firstcutaneous actuator 1117 occurs temporally prior to the haptic output forthe second cutaneous actuator 1118. The attack 1123 portion of thehaptic output for the second cutaneous actuator 1117 occurs temporallyafter the attack 1119 portion of the first cutaneous actuator by astimulus onset asynchrony (SOA) 1122 delay period. In one embodiment,this SOA is 0.17×D+65, where D refers to the length of the duration 1120of the haptic output for the first cutaneous actuator 1117 and both 0.17and 65 are constants which modify the duration value. Thus, the SOAperiod is a function of the duration of the haptic output. The result ofthe two different haptic outputs occurring at two different cutaneousactuators is that the user experiences a motion between the twoactuators that appears to be a smooth stroking motion between the twoactuators, and specifically from the first cutaneous actuator 1117 tothe second cutaneous actuator 1118.

In one embodiment, this “two point” haptic output represents a vowelphoneme of an input word 1106. Thus, if a vowel phoneme is encounteredin an input word, the corresponding sequence of actuator signals 1111which is outputted causes two different cutaneous actuators to generatetwo different haptic outputs separated by a temporal delay that may besimilar to the one shown here in FIG. 11D.

FIG. 11E illustrates an example of haptic outputs of differentdurations, according to an embodiment. In addition to the single and twopoint haptic outputs shown in FIG. 11C and FIG. 11D, respectively, wherethe duration shown for the haptic outputs is the same, in otherembodiments the haptic output generated by a cutaneous actuator 1104from a actuator signal 1111 can be of different durations. For example,as illustrated in FIG. 11E, one of the haptic outputs with attack 1125and decay 1127 has a longer duration 1126 (e.g., 400 ms) where theintensity of the haptic output is at a maximum level (i.e., after theattack 1125 and before the decay 1127), and the other haptic output withattack 1128 and decay 1130 has a relatively shorter duration 1129 (e.g.,150 ms) where the haptic output is at the maximum level. Like the singleand two point haptic outputs, the haptic outputs illustrated in FIG. 11Ewith different durations may be used to represent different phonemes ofthe input words 1106. For example, while a two point haptic output mayrepresent a short vowel phoneme, while a two point haptic output witheach haptic output having longer duration may be used to represent along vowel phoneme.

FIG. 11F illustrates an example of haptic outputs of differentfrequencies, according to an embodiment. In addition to the single, twopoint, and different duration haptic output described above, in otherembodiments the haptic output generated by a cutaneous actuator 1104from a actuator signal 1111 can have a repeating pattern, such as awaveform, with the repetition of the pattern being at differentfrequencies. FIG. 11F illustrates a low frequency 1131 pattern and ahigh frequency 1133 pattern. The high frequency 1133 pattern repeats ata higher frequency than the low frequency pattern 1131. In theillustrated example, the duration 1132 of the low frequency pattern isalso longer than the duration 1134 of the high frequency 1133 pattern,however, this does not always need to be the case. If the haptic outputis a vibration that can be represented as a waveform, then the durationof each pattern in the waveform can be considered a wavelength, and themaximum intensity of the haptic output can be considered the maximumamplitude. Furthermore, although a single maximum intensity level forthe haptic output is illustrated for both the low frequency 1131 andhigh frequency 1133 haptic outputs, in other embodiments the amplitudecan be variable and may differ depending on the frequency, and mayfurther differ over time.

In one embodiment, the different frequencies of haptic output may betargeted towards different sensory cells in the user's skin. In oneembodiment, the higher frequency haptic outputs (e.g., 250 Hz), such asthe high frequency 1133 haptic output, are targeted towards Paciniancorpuscle cells, while the low frequency haptic outputs (e.g., 30 Hz),such as the low frequency 1131 haptic output, are targeted towardsMerkel cells. In such a case, the frequency of the high frequency 1133haptic output is sufficiently high to trigger response in the Paciniancorpuscle cells and to avoid triggering the Merkel cells, whereas thelow frequency 1131 haptic output is of a frequency that triggers theMerkel cells but not the Pacinian corpuscle cells. As Pacinian corpusclecells (also known as Lamellar corpuscle cells) are sensitive tovibration, while Merkel cells are sensitive to general touch, differenttypes of haptic output can target different mechanoreceptor cells in theuser's skin and be associated with different phonemes when convertingthe input words 1106 to actuator signals 1111 which are used to causethe cutaneous actuators 1104 to generate the haptic outputs.

In one embodiment, the low frequency 1131 and high frequency 1133 hapticoutputs are generated by a single cutaneous actuator of the cutaneousactuators 1104, but with a linear offset between the two haptic outputs,which may cause a difference between the phase of the two differenthaptic outputs. Thus, the haptic output of the single cutaneous actuatormay appear to resemble a summation of the high frequency 1133 and lowfrequency 1131 haptic outputs, but with an offset between the lowfrequency 1131 and high frequency 1133 haptic outputs (e.g., 10% of theduration or frequency). Despite having combined the two haptic outputs,because the user's skin senses the high and low frequency haptic outputsusing different cells, there may be little ambiguity when the usersenses the haptic output and thus the user may be able to easilydistinguish the two different frequencies in the haptic output. Thedifferent frequencies of haptic output may represent different phonemesin the input words 1106. For example, high frequency haptic output mayrepresent digraphs (e.g., “/

/ or zh) while the low frequency haptic output may represent certainconsonants.

FIG. 11G illustrates an example of haptic outputs of differentamplitudes, according to an embodiment. In addition to the varioushaptic output types shown above, in other embodiments the haptic outputgenerated by a cutaneous actuator 1104 from a actuator signal 1111 canbe of different intensities, otherwise known as amplitudes. For example,as illustrated in FIG. 11G, the three different haptic outputs haveamplitudes 1136, 1137, and 1138. These haptic outputs with differentamplitudes may be used to represent different phonemes or sounds of theinput words 1106. For example, a larger amplitude of haptic output mayrepresent an emphasis, end of sentence, punctuation mark, or so on,while a smaller amplitude may represent normal speech.

While certain types of haptic outputs have been described above withreference to FIGS. 11C through 11G, in other embodiments additionalhaptic output types may be generated by the cutaneous actuators 1104.Furthermore, the haptic outputs described above may represent more thanjust phonemes, and may represent numbers, punctuation marks, tone ofvoice, musical notes, audio cues, and so on. In addition, the sequenceshaptic outputs described above may be generated by more than onecutaneous actuator 1104 to represent a single phoneme or other element.

FIG. 11H illustrates exemplary sequence of haptic outputs for an exampleinput word 1106, according to an embodiment. In the illustrated example,the input word 1106 is “patent.” This word has 6 different phonemes: 1)/p/, 2) /ā/, 3) /t/, 4) /

/, 5) /n/, and 6) /t/. Note that the second and fourth phonemes arevowels. The word-haptic signal converter 1107 converts each of thesephonemes into a sequence of actuator signals 1111, which are transmittedto the cutaneous actuators 1104, which generate a haptic output, such asa vibrational output, based on the actuator signals 1111 and which aresensed by the user. For ease of discussion, each of the sequence ofhaptic outputs associated with a phoneme is referred to as a hapticphoneme. Thus, the six phonemes of the word “patent” are translated intosix haptic phonemes which are sensed by the user. After training, theuser may be able to identify the word “patent” after sensing these sixhaptic phonemes. A similar process would occur for other words.

In the illustrated example of FIG. 11H, a 3×2 two dimensional array ofcutaneous actuators 1161-1166 is represented by concentric circles, witheach set of two concentric circles representing one cutaneous actuator.The cutaneous actuators 1161-1166 are labeled for haptic phoneme 1139,but are not repeatedly labeled for haptic phonemes 1140-1144 forclarity. However, the cutaneous actuators indicated in haptic phonemes1140-1144 are the same as those in the same positions in haptic phoneme1139. This two dimensional array of cutaneous actuators 1161 through1166 may correspond to the two dimensional array of cutaneous actuators1104 in FIG. 11A.

Each haptic phoneme 1139-1144 indicates a sequence of haptic outputs bythis array of cutaneous actuators that corresponds to a phoneme of theinput word “patent”. Each single haptic output in a sequence of hapticoutputs corresponds to activation of a cutaneous actuator, and isrepresented by a set of two solid concentric circles for that cutaneousactuator. In contrast, non-activated cutaneous actuators are representedby dotted concentric circles. Furthermore, a two point sequence ofhaptic outputs is represented by two sets of solid concentric circles,with an arrow in between the two sets indicating an activation of afirst cutaneous activator which is located in a direction immediatelyopposite the direction of the arrow, followed by the activation of asecond cutaneous actuator which is indicated in a direction immediatelyfollowing the direction of the arrow. The two point sequence of hapticoutputs may be similar to the sequence described above with reference toFIG. 11D. For example, for haptic phoneme 1140 for the phoneme “/ā/” isindicated by the activation of the cutaneous actuator 1161 followed bythe activation of the cutaneous actuator 1163 (after a delay period).

The sequences of haptic outputs for each haptic phoneme may be separatedfrom each other by a delay period with no haptic output (e.g., of 200ms), or by a special delimiter haptic output (e.g., all actuatorsactivate at the same time for an interval of time).

In the illustrated example, 1) /p/ corresponds to the haptic phoneme1139 and activation of the cutaneous actuator 1166, 2) /ā/ correspondsto the haptic phoneme 1140 and activation of the cutaneous actuator 1161followed by 1163, 3) /t/ corresponds to the haptic phoneme 1141 andactivation of the cutaneous actuator 1164, 4) /

/ corresponds to the haptic phoneme 1142 and activation of the cutaneousactuator 1165 followed by 1166, 5) /n/ corresponds to the haptic phoneme1143 and activation of the cutaneous actuator 1163, and 6) the second/t/ corresponds to the haptic phoneme 1144 and activation of thecutaneous actuator 1164 (same as the first /t/).

The correspondence between each haptic phoneme and phoneme may beselected based on articulation association. This means that acorrespondence is made between the physical location of the hapticoutput of each haptic phoneme in the two dimensional array of cutaneousactuators and the location of articulation of the corresponding phonemewhen spoken by a human. For example, the phoneme /p/ is the voicelessbilabial stop, which corresponds to an articulation of the user's lips.This may correspond to a location in the two dimensional array ofcutaneous actuators representing the front of a user's mouth. In theillustrated example, this is cutaneous actuator 1166, which is activatedin response to the phoneme /p/. The sequences of haptic outputs for theremaining haptic phonemes may be constructed similarly.

Furthermore, in one embodiment, the duration of haptic output forconsonant phonemes may be shorter than that of vowel phonemes. This mayoccur either due to the shorter duration for each activation of acutaneous actuator for a consonant phoneme versus a vowel phoneme, ordue to the vowel phoneme using a two point sequence of haptic outputversus a single haptic output for a consonant phoneme. For example, ahaptic output for a consonant phoneme may be of a duration of 180 ms,while a haptic output for a vowel phoneme may be of a duration of 270.5ms. For consonant phonemes, the duration of the haptic output may bedetermined experimentally by starting at a baseline duration (e.g., 150ms) and adding the time needed to “ramp up” and “ramp down” (asdescribed above), which may be 20% of the duration time. For vowelphonemes, the duration of the haptic output may be determined based on aseparate baseline (e.g., 100 ms), plus the ramp up and ramp downinterval, and plus the SOA interval.

The illustrated array of cutaneous actuators may allow for a combinationof 36 haptic phonemes, which includes all combinations of single hapticoutputs (6) and two point haptic outputs (30). The two point hapticoutputs are limited to those that 1) do not involve activation of asecond cutaneous actuator which is diagonal from the first activatedcutaneous actuator; and 2) which are short distance motions, i.e., thefirst and second activated cutaneous actuators are situated adjacent toeach other. However, this may not represent the total number of phonemesin a language. In such a case, the number of cutaneous actuators may beincreased to accommodate for the total number of phonemes. An example ofsuch a combination of cutaneous actuators is shown and described withreference to FIG. 11I. Note that although a particular combination ofhaptic outputs are shown here for each phoneme, in other embodiments thecombinations of haptic outputs to phonemes may differ.

FIG. 11I illustrates an alternate arrangement of haptic devices witharrays of cutaneous actuators, according to an embodiment. As notedabove, in some cases there may be more phonemes or other speech elementsthat need to be represented via haptic output but an insufficient numberof combinations of haptic outputs available with a particular array ofcutaneous actuators. Additionally, in some cases the use of the entireavailable set of combinations of haptic outputs for an array ofcutaneous actuators may be undesirable, as some combinations of hapticoutputs may be too similar to other combinations, resulting in a higherror rate (i.e., low disambiguation rate) for users of the hapticdevice. Thus, more cutaneous actuators may be needed in these scenarios.The device shown in FIG. 11I is an example of a device with additionalcutaneous actuators. This device includes a forearm haptic device 1145with two 4×2 array of cutaneous actuators 1146 on the dorsal side andthe ventral side of the user's forearm, as well as an upper arm hapticdevice 1147 with a set of 4×2 array of cutaneous actuators 1148 on thedorsal side of the user's upper arm. The combination of these two hapticdevices allows for 24 individual combinations of single haptic outputs(via the 24 individual cutaneous actuators), as well as a largecombination of two point haptic outputs. This system supports a muchlarger number of haptic phonemes for translation of phonemes in inputwords 1106 to actuator signals 1111 and to haptic outputs in thecutaneous actuators.

In addition to the alternate haptic device shown in FIG. 11I, thecutaneous actuators used to deliver the haptic output could be attachedto an arm band, body armor, gloves, a jacket, a shirt, vest, sole of ashoe, on a wearable device, on a smartphone, and so on, so long as thecutaneous actuators on these devices can deliver a haptic output to theuser.

FIG. 11J is a flowchart illustrating a method of converting input wordsinto haptic output based on phonemes, according to an embodiment. Theprocess described in the flowchart may be performed by the word-hapticsignal converter 1107, which in turn may be a component of a signalgenerator that may be part of a haptic device. Although a particulararrangement of steps is shown here, in other embodiments the process maybe arranged differently.

Initially, the word-haptic signal converter 1107 receives 1150 an inputword which is a unit of a language, such as English. The word-hapticsignal converter 1107 converts 1151 the input word into its componentphonemes. For example, the word “apple” would be converted into thephonemes “ae,” “p,” “ah,” and “l.”

The word-haptic signal converter 1107 converts 1152 the phonemes into asequence of actuator signals. This sequence is formed from aconcatenation of sub-sequences of actuator signals. Each phonemecorresponds to one of these sub-sequences of actuator signals. Each ofthe actuator signals are mapped to a cutaneous actuator of a twodimensional array of cutaneous actuators placed on an interior surfaceof a harness shaped to wrap around a user's arm. The interior surface ofthe harness facing the user's arm when worn by the user.

The word-haptic signal converter 1107 transmits the sequence of actuatorsignals to the two dimensional array of cutaneous actuators. This causesthe cutaneous actuators to activate in accordance with the sequence ofactuator signals, and in turn, causes the user to sense a series ofhaptic outputs from the cutaneous actuators. If trained, the user maythen be able to recognize the haptic outputs as phonemes of a word, anddecipher the input word from the haptic outputs. In this fashion, a usermay be able to recognize words haptically rather than aurally orvisually.

FIG. 11K illustrates an exemplary mapping of the phonemes “/p/,” “/b/,”“/t/,” “/v/,” “/f/,”, and “/θ/,” into sequences of haptic outputs,according to an embodiment. In addition to the phonemes to hapticoutputs shown in FIG. 11K, FIGS. 11L-11Q illustrate additional exemplarymappings between phonemes and haptic outputs. Additionally, FIGS.11R-11T illustrate an exemplary rationale for the mapping of thephonemes to haptic outputs of FIGS. 11K-11Q based on place and manner ofarticulation. In FIGS. 11K-11Q, representations of a user's arm areshown. The arm includes multiple cutaneous actuators, represented bycircles. As shown, an array 1167 of cutaneous actuators is present onthe posterior (“back side”) of the user's forearm, another array 1168 ofcutaneous actuators is present on the anterior of the user's forearm,and yet another array 1169 of cutaneous actuators is present on theposterior of the user's upper arm. The location of these arrays ofcutaneous actuators may correspond to those shown in FIG. 11I, with thearrays 1167 and 1168 corresponding to the cutaneous actuators in theforearm haptic device 1145, and the array 1169 corresponding to thecutaneous actuators of the upper arm haptic device 1147. Each array1167-1169 includes a set of 4×2 (or 8 total) cutaneous actuators, withthe cutaneous actuators arranged lengthwise along the user's arm, witheach array having four rows of two cutaneous actuators each arrangedalong the proximal-distal axis of the user's arm.

FIG. 11K additionally illustrates a legend 1170. The legend 1170indicates a type of haptic output corresponding to an illustrated fillpattern of some of the circles that represent the cutaneous actuators. Acircle with a fill pattern indicates that the cutaneous actuatorcorresponding to the circle is by itself or in combination with othercutaneous actuators generating the type of haptic output indicated inthe legend for that fill pattern. The indicated types of haptic outputs,as numbered in the legend 1170, are 1) a continuous 250 Hz sine waveoutput, 2) a pulsatile output, 3) a fast motion, 4) a vibration pluspressure, and 5) a vibration plus shear. The first listed type output,the continuous 250 Hz sine wave, corresponds to a sine wave movement ata cutaneous actuator of 250 Hz. This means that the cutaneous actuatorchanges the haptic output smoothly over time, in accordance with theshape of a sine wave. The (second listed) pulsatile output correspondsto a repeated pattern of activation to a specific activation position ofa cutaneous actuator for a period of time, followed by a de-activationof the cutaneous actuator for a period of time. The activation anddeactivation a similar to a square wave, in that the cutaneous actuatoractivates to the specified activation position as quickly as it iscapable, and similarly deactivates as quickly as possible. As thecutaneous actuator may have a response time for activation anddeactivation, the haptic output may not represent a perfect square wave,but rather a trapezoidal waveform. The fast motion output is similar tothe output indicated in 1), but operates at twice the frequency, i.e.,500 Hz. Note that although 250 Hz and 500 Hz are indicated here, otherfrequencies may also be used instead. The vibration plus pressure outputcorresponds to an output that has both a vibratory sensation and apressure sensation. The vibratory sensation may be achieved by a sinewave output similar to the output type 1) with a frequency above athreshold that could be perceived as vibration (e.g., exceeding 100 Hz)or with the output similar to the output type 2), or any other type ofoutput that provides a rapid and continuous change to the haptic outputthat may be sensed as vibration. The pressure sensation is a sensationat the user's arm of a pressure that exceeds a normal pressure (i.e.,where the cutaneous actuator is not activated). The pressure sensationmay be achieved by having the cutaneous actuator activate beyond athreshold level, such that the cutaneous actuator applies a minimumlevel of pressure to the user's arm. The cutaneous actuator may applythis minimum pressure in addition to the output that generates thevibratory sensation. The vibratory sensation and the pressure sensationmay be applied simultaneously or sequentially. The vibration plus shearoutput corresponds to an output that has a vibratory sensation and ashear sensation (which may be applied simultaneously or sequentially).The vibratory sensation may be similar or identical to the vibratorysensation described above for the vibration+pressure output. The shearsensation is a sensation of a movement that is tangential to the surfaceof the user's arm. This may be achieved by a smooth transition betweenactivations of multiple cutaneous actuators, resulting in a sensation ofa sliding movement along the path of the sequentially activatedcutaneous actuators. Alternatively, a cutaneous actuator may be able togenerate, by itself, a shear movement.

The haptic output indicated by the fill patterns for each of thedepictions of the user's arms in FIGS. 11K-11Q correspond to a phoneme,which is shown as the underlined portion of the sample word that isadjacent to the respective arm. For example, phoneme 1171 is the phoneme“/p/” (using IPA notation). This phoneme is present in the pronunciationof the word “pay.” Therefore, the word “pay” is shown, with the letter“p,” corresponding to the phoneme “/p/,” underlined. A similar method ofindicating the phoneme is provided for the remaining phonemesillustrated in FIGS. 11K-11Q. Additionally, some of the phonemes shownin FIGS. 11K-11Q have arrows which indicate the sequence of activationsof the cutaneous actuators that generate the sequence of haptic outputsthat are mapped to each phoneme. In some cases, such as with thevibration plus shear output, multiple cutaneous actuators may beactivated in sequence to generate the desired type of haptic output. Inthese cases, the arrows not only indicate the sequence of hapticoutputs, but also how the specified type of haptic output is achieved.For example, in the case of vibration plus shear, the arrow indicatesthe direction of the shear sensation. Furthermore, when a sequence ofcutaneous actuators are activated, one cutaneous actuator may deactivatebefore the activation of the next cutaneous actuator in the sequence, orthe activations of each may overlap by a specified amount. For example,one cutaneous actuator may begin the process of deactivation while thenext cutaneous actuator in sequence may begin the activation process.Additional details regarding the different sequences of haptic outputscorresponding to various phonemes are described below with reference toFIGS. 11K-11Q. Note that the IPA-transcribed phonemes described belowfor the sample words correspond to General American Englishpronunciation.

FIG. 11K illustrates examples of haptic outputs for the phonemes “/p/,”“/b/,” “/t/,” “/v/,” “/f/,” and “/θ/.” These are labeled, in the sameorder, as phonemes 1171-1176 in FIG. 11K. As noted above, the phonemesare labeled as an underlined portion of an exemplary word containingthat phoneme. For example, phoneme 1176 uses the exemplary word “theta,”with the “th” being underlined, corresponding to the phoneme “/θ/” (i.e.the voiceless dental fricative).

Thus, the phoneme 1171 or “/p/” is represented by a single activation ofa cutaneous actuator using a continuous sine wave output (type 1) asshown, and corresponds to the “p” in the sample word “pay” as shown. The“/b/” phoneme 1172 is represented by a pulsatile output (type 2) of asingle cutaneous actuator as shown, and corresponds to the “b” in “bee.”The “/t/” phoneme 1173 is represented by a single cutaneous actuatorwith a continuous sine wave output (type 1) as shown, and corresponds tothe “t” in “tea.” The “/v/” phoneme 1174 is represented by a vibrationplus shear output (type 6) with a single cutaneous actuator as shown,and corresponds to the “v” in “victor.” The “/f/” phoneme 1175 isrepresented by a vibration plus pressure output (type 5) with a singlecutaneous actuator as shown, and corresponds to the “f” in “fee.” The“/θ/” phoneme 1176 by a vibration plus pressure output (type 5) with asingle cutaneous actuator as shown, and corresponds to the “th” in“theta.”

FIG. 11L illustrates examples of haptic outputs for the phonemes “/d/,”“/k/,” “/g/,” “/ð/,” “/s/,” and “/z/,” according to an embodiment. Asillustrated, the phoneme “/d/” corresponds to the “d” in “day,” “/k/” tothe “c” in “cake,” “/g/” to the “g” in “go,” “/ð/” (a voiced dentalfricative) to the “th” in “thee,” “/s/” to the “s” in “see,” and “/z/”to the “z” in “zee.” As with FIG. 11K, each phoneme is represented by acorresponding activation of one or more cutaneous actuators generating ahaptic output of a type indicated by the fill pattern as shown. Forclarity and brevity's sake, a description of the different hapticoutputs will not be repeated here for each of the different phonemes formost of the remainder of the phonemes shown in FIGS. 11L-11Q, unlessspecial distinction is needed. Instead, reference is made to the figureitself which describes the different types of haptic outputscorresponding to each phoneme.

FIG. 11M illustrates examples of haptic outputs for the phonemes “/∫/,”“/

/,” “/h/,” “/n/,” “/

/,” and “/l/,” according to an embodiment. As illustrated, the phoneme“/∫/” (a voiceless postalveolar fricative) corresponds to the “sh” in“she,” “/

/” (a voiced palatoalveolar sibilant fricative) to the “z” in “seizure,”“/h/” to the “h” in “he,” “/n/” to the “n” in “name,” “/

/” (a velar nasal) to the “ng” in “ping,” and “/l/” to the “l” in “lee.”As with FIG. 11K, each phoneme is represented by a correspondingactivation of one or more cutaneous actuators generating a haptic outputof a type indicated by the fill pattern as shown. Note that the phoneme“/l/” here corresponds to a haptic output that comprises sequentialactivation of four different cutaneous actuators, in the order shown bythe circular arrow illustrated in FIG. 11M for the phoneme “/l/” and asrepresented by the sample word “lee.”

FIG. 11N illustrates examples of haptic outputs for the phonemes “/

/,” “/d/,” “/m/,” “/

/,” “/w/,” and “/j/,” according to an embodiment. As illustrated, thephoneme “/

/” (a voiceless palato-alveolar affricate) corresponds to the “ch” in“cheese,” “/d/” to the “g” in “gym,” “/m/” to the “m” in “my,” “/

/” (an alveolar approximant) to the “r” in “read,” “/w/” to the “w” in“we,” and “/j/” to the “y” in “yield.” As with FIG. 11K, each phoneme isrepresented by a corresponding activation of one or more cutaneousactuators generating a haptic output of a type indicated by the fillpattern as shown. Note that the phonemes “/

/,” “/d/,” and “/w/,” all correspond to haptic outputs of multiplecutaneous actuators as indicated by the straight arrow, with directionof activation sequence, shown above the illustrations of their hapticoutputs as shown in FIG. 11N. For example, the phoneme “/d/” shows anactivation of cutaneous actuators from the proximal to distal positionsalong a single column of cutaneous actuators, as shown.

FIG. 11O illustrates examples of haptic outputs for the phonemes “/i/,”“/I/,” “/e/,” “/æ/,” and “/ε/,” according to an embodiment. Asillustrated, the phoneme “/i/” corresponds to the “ee” in “deed,” “/I/”(a near-close near-front unrounded vowel) to the “i” in “sit,” “/e/” tothe “ay” in “may,” “/æ/” (an near-open front unrounded vowel) to the “a”in “at,” and “/ε/” (an open-mid front unrounded vowel) to the “e” in“set.” As with FIG. 11K, each phoneme is represented by a correspondingactivation of one or more cutaneous actuators generating a haptic outputof a type indicated by the fill pattern as shown. Note that the phoneme“/æ/” is represented by a sequence of haptic outputs on the anterior ofthe user's forearm (i.e., in the array 1168).

FIG. 11P illustrates examples of haptic outputs for the phonemes “/

/,” “/Λ/,” “/

/,” “/

/,” and “/u/,” according to an embodiment. As illustrated, the phoneme“/

/” (an r-colored open-mid central unrounded vowel) corresponds to the“ur” in “hurt,” “/Λ/” (an open-mid back unrounded vowel) to the “u” in“but,” “/

/” (an open back rounded vowel) to the “o” in “odd,” “/

/” (an near-close near-back rounded vowel) to the “u” in “bull,” and“/u/” to the “oo” in “moon.” As with FIG. 11K, each phoneme isrepresented by a corresponding activation of one or more cutaneousactuators generating a haptic output of a type indicated by the fillpattern as shown.

FIG. 11Q illustrates examples of haptic outputs for the phonemes “/

/,” “/

/,” “/aI/,” “/

I/,” and “/a

/,” according to an embodiment. As illustrated, the phoneme “/

/” (a mid-central unrounded vowel) corresponds to the “o” in “oat,” “/

/” (an open-mid back rounded vowel) to the “au” in “auto,” “/aI/” to the“i” in “I,” “/

I/” to the “oy” in “toy,” and “/a

/” to the “ow” in “cow.” Note that the last three phonemes describedhere are diphthongs. As with FIG. 11K, each phoneme is represented by acorresponding activation of one or more cutaneous actuators generating ahaptic output of a type indicated by the fill pattern as shown.

FIG. 11R is a chart illustrating the consonant phonemes in FIGS.11K-11Q, and the mapping between the location, duration, and type of thehaptic outputs corresponding to each consonant phoneme and the manner ofarticulation and place of articulation of each consonant phoneme,according to an embodiment. The row legend describes the differentmanners of articulation of the phonemes described in FIGS. 11K-11Q,while the column legend describes the place of articulation of the samephonemes. The manner of articulation is the configuration andinteraction of the articulators, which include speech organs such as thetongue, lips, and palate, when making a speech sound, i.e., the phoneme.The place of articulation is the point of contact where an obstructionoccurs in the vocal tract between an articulatory gesture (e.g., a mouthmovement), an active articulator (e.g., a part of the tongue), and apassive location (e.g., the roof of the mouth). These apply to consonantphonemes.

In one embodiment, the location and type of the haptic output associatedwith each phoneme, as described above in FIGS. 11K-11Q, have arelationship to the manner of articulation and place of articulation ofthat phoneme. Thus, the haptic outputs may be associated with thephonemes in a manner that adheres to certain rationale in order tocreate this relationship. These rationale are described below.

1) Consonant phonemes may be related to haptic outputs that are at asingle point, such that no movement (e.g., shear) is sensed by the userdue to the haptic output.

2) In contrast, vowel consonants may be related to haptic output thatcreate a sense of motion for the user. This may include a haptic outputor sequence of haptic outputs that have a shear sensation.

3) Shorter vowels may have a total duration of haptic output that isrelatively shorter than the total duration of haptic output related tolonger vowels.

4) Voiced phonemes (those that use the larynx) are related to hapticoutputs having a pulsatile type.

5) Unvoiced or voiceless phonemes are related to haptic outputs having avibration type.

6) Phonemes that are produced closer to the front of the mouth (orgenerally closer to the mouth) are mapped to haptic outputs that arecloser to the user's wrist.

7) Phonemes that are produced closer to the rear of the throat (orgenerally closer to the throat) are mapped to haptic outputs that arecloser to the user's shoulder.

8) Plosives (i.e., a stop, or a consonant phoneme which blocks airflowin the vocal tract) are related to a haptic output that is at a singlepoint or location on the user's upper arm.

9) Fricatives (i.e., consonants that force air through a narrow channel,such as lips, tongue/soft palate, etc.) are related to a haptic outputthat includes two points of haptic output on the user's upper arm thatare of a vibration type.

10) Nasals (i.e., a phoneme made with the soft palate lowered so thatsound escapes through the nose) are related to a haptic output of asingle point on the user's forearm.

Using the above rationale, a mapping between phonemes and haptic outputscan be generated, such as the one shown in the above FIGS. 11K-11Q. FIG.11R further illustrates this mapping between haptic outputs and phonemesby showing the relationship between the characteristics of the hapticoutputs and the manner and place of articulation of the correspondingmapped phonemes. These characteristics may include the location of thehaptic output on the user's arm, the duration, the type, the movementexperienced, the sequence of cutaneous actuators activated, and so on.For example, the manner of articulation of the phonemes “/p/” and “/b/”are voiceless and voiced stop consonants. Furthermore, the place ofarticulation of the phonemes “/p/” and “/b/” are bilabial, or near thelips. Thus, the corresponding mapping of these phonemes is to hapticoutputs that are 1) at a single point, as they are consonants, 2) nearthe user's wrist, as they are plosive, and 3) vibration type for thevoiceless “/p/” and pulsatile for the voiced “/b/.” The remainingconsonant phonemes shown in FIG. 11R are mapped in a similar fashion.

For the places of articulation listed, labiodental consonants arearticulated with the lower lip and upper teeth, and dental consonantsare articulated with the tongue against the upper teeth. Both thesecorrespond to haptic outputs near the user's wrist. Alveolar consonantsare articulated with the tongue against/close to the superior alveolarridge, which is between the upper teeth and the hard palate, and palatalconsonants are articulated with the body of the tongue raised againstthe hard palate (the middle part of the roof of the mouth). Both thesetypes of consonants are mapped to haptic output that have locationsmidway between the user's shoulder and the user's wrist on the hapticdevice shown FIGS. 11K-11Q. Velar consonants are articulated with theback part of the tongue against the soft palate (the back of the roof ofthe mouth, or velum). Glottal consonants use the glottis (i.e., theopening between the vocal folds) for articulation. Thus, these aremapped to haptic output that is nearer the user's shoulder on the user'supper arm.

For the manners of articulation listed, fricatives and nasals, alongwith their corresponding mappings, have been noted above. Affricateconsonants begin as a stop or plosive and release as a fricative, andmap here to vibration plus a linear shear type movement for the hapticoutput. Liquid consonants include lateral consonants (in English—“l” in“led”) and rhotics (in English—“r” in “red”). These are mapped to hapticoutputs that have a vibration type plus a shear type that has a circularactivation of the cutaneous actuators resulting in a circular shearsensation. Finally, glide, or semivowel consonants, are phonemes thatare similar to a vowel sound but do not act as the nucleus of asyllable, such as the “w” and “y” consonants. These are mapped tovibration plus pressure haptic outputs with an additional sequence ofactivation of cutaneous actuators as indicated by the respective arrows.For these cases, the level of pressure generated may increase with eachsuccessive cutaneous actuator that is activated.

FIG. 11S is a chart illustrating the vowel phonemes in FIGS. 11K-11Q,and the mapping between the location, duration, and type of the hapticoutputs corresponding to each vowel phoneme and the manner ofarticulation and place of articulation of each vowel phoneme, accordingto an embodiment. The illustration in FIG. 11S is similar to that ofFIG. 11R, but is noted for vowel phonemes instead of the consonantphonemes in FIG. 11S. Here, vowels are divided into their place ofarticulation: 1) front, 2) central, and 3) back, describing the positionof the tongue in the mouth on the transverse plane (i.e., front/anteriorof mouth, center of mouth, or back/posterior of mouth) in the formationof the phoneme. The vowels are further divided into 1) high, 2) mid, and3) low vowels, describing the position of the tongue in the mouth on thefrontal plane (i.e., near the roof of the mouth, in-between, or near thebottom of the mouth) in the formation of the vowel. As noted above inthe list of rationales, the position of the tongue indicates theappropriate position of the haptic output for a phoneme. Thus, the “/

/” vowel phoneme in “odd” is a central and low vowel, meaning the tongueis nearer to the user's throat, and thus the corresponding haptic outputis on the user's upper arm. Note also that some of the haptic outputs,such as the “/u/” in “moon” activate a sequence of haptic outputs thattransitions from one array of haptic outputs on one side of the user'sforearm to the other side. Such a sequence may be mapped for vowels withcertain places of articulation, such the back vowels shown here.

FIG. 11T is a chart illustrating the diphthong phonemes in FIG. 11Q, andthe mapping between the location, duration, and type of the hapticoutputs corresponding to each diphthong phoneme and the manner ofarticulation and place of articulation of each diphthong phoneme,according to an embodiment. Diphthongs, or gliding vowels, are acombination of two adjacent vowels sounds within the same syllable.These diphthongs are mapped to sequences of the type 1) haptic outputdescribed above (sine wave vibration), with the start and end locationof the sequence of haptic output matching the place of articulations ofthe tongue of the starting and ending vowels for the diphthong. Forexample, the diphthong “oy” in the word “toy” begins with the vowelphoneme “/

/,” which is a back vowel, and ends as a front vowel phoneme /I/, asshown by the arrow in FIG. 11S which emanates from the word “cow” andpoints towards the front vowel section. Thus the haptic output sequencemapped to this diphthong begins nearer the shoulder, and ends nearer thewrist. A similar mapping is performed for the “I” diphthong and “ow”diphthong in the word “cow,” as shown in FIG. 11S.

Calibration of Haptic Device Using Sensor Harness

Embodiments also relate to a method for calibrating a haptic device toprovide haptic outputs that are consistently perceived across differentusers in accordance to each user's subjective sensation. Each user'ssubjective experience due to force applied to the user's skin may vary.Due of these subjective differences, a haptic feedback system wouldcreate inconsistent experiences across users if it were to apply thesame objective level of haptic outputs for all users. A haptic feedbacksystem is calibrated on a per-user basis such that each user experiencesthe same subjective level of haptic outputs as other users.

FIG. 12A illustrates a haptic device 1200 that may be attached to auser's forearm 1202 to calibrate a haptic device based on subjectiveforce, according to an embodiment. The haptic device 1200 includes anadjustable harness 1204, and one or more sensors 1206.

The sensors 1206 detect one or more types of displacement or parametersassociated with haptic output including acceleration, strain, and so on.These sensors may be similar to the haptic sensors 104 described abovewith reference to FIG. 1. They are attached to the interior surface ofthe adjustable harness 1204 at regular intervals and face the skin ofthe user's forearm 1202 (i.e., the sensing components of the sensors1206 face the user's skin). The sensors 1206 are separated by at least athreshold distance from each other. In one embodiment, there are threesensors 1206 total. After the adjustable harness 1204 is fitted on theuser, the user's skin may contact the sensors 1206 and cause the sensorsto detect a force value, which may be in the form of a voltage valuecorresponding to the force value. This voltage value is transmitted to acalibration component, such as that described with reference to FIG.12B.

The adjustable harness 1204 may be similar to the harness 1103 of FIG.11A, and may use similar materials and have similar structuralcomponents, including, for example, rigid or non-rigid members thatcause the adjustable harness 1204 to be worn in relatively similarpositions on the user's forearm 1202 each time it is fitted to theuser's forearm 1202. However, the adjustable components of theadjustable harness 1204 may allow for a more fine-grained adjustment ofthe fitment of the adjustable harness 1204 against the user's forearm.In particular, in one embodiment, the adjustable harness 1204 includesan adjustment mechanism (not shown) at the location of each sensor 1206.This adjustment mechanism allows for adjustment of the haptic outputexerted by the adjustable harness 1204 against the user's forearm at thelocation of the sensors 1206 by adjusting the circumference of theadjustable harness 1204, or via some other means. Furthermore, theadjustment mechanism may have various markings to indicate the level ofadjustment that has occurred. For example, form of markings thatindicate a dimension of the adjustable harness 1204 (e.g., acircumference of the adjustable harness 1204) at a particular level ofadjustment may be provided. An adjustment mechanism may have markingindicating the total circumference of the adjustable harness 1204 ateach level of adjustment. The adjustment mechanism may be an adjustablestrap (which may be elastic or non-elastic), a hook and loop fastenercombination (e.g., Velcro®), a retractable strap, a ratcheting strap orbuckle, and so on.

The user, after wearing the adjustable harness 1204, can use theadjustment mechanisms to adjust the haptic output provided by theadjustable harness 1204 on the user's forearm 1202 to a certain levelthat is comfortable to the user, or in the case of calibration, to alevel that achieves a specified subjective level of the haptic output.By adjusting the adjustable harness 1204 to achieve a specificsubjective level of the haptic output, the sensors 1206 may be able todetermine the objective level of the haptic output generated by theadjustable harness 1204 at a particular adjustment position (asdescribed above). This information is transmitted to an additionalsystem that can determine a correlation between the user's subjectivelevel of the haptic output (i.e., the force that is felt) versus theactual objectively measured level of the haptic output based on thesensor 1206 output. Such information can be used to calibrate theresponse of haptic feedback devices for each user, such that thesubjective level of the haptic output generated by these haptic feedbackdevices are the same for each user. Additional details regarding thissystem and method of calibration will be described in further detailwith reference to FIGS. 12B through 12D.

FIG. 12B illustrates detail views of two different examples of thehaptic device 1200 of FIG. 12A, according to an embodiment. Two types ofadjustable harnesses for the haptic device 1200 are shown in FIG.12B: 1) the adjustable harness 1228; and 2) the adjustable harness 1236.

The adjustable harness 1228 may include, among other components, threepressure sensors 1226A-C, each attached to the adjustable harness 1228.At each point on the adjustable harness 1228 at which a pressure sensor1226 is attached, an adjustable strap 1230 is attached to the adjustableharness 1228. The adjustable strap 1230 may be an adjustable mechanismas described above with reference to FIG. 12A. The adjustable strap 1230adjusts the force of the pressure sensors 1226 against the forearmsurface 1232. This may be accomplished by tightening or loosening theadjustable harness 1228 against the forearm surface 1232, which mayreduce the circumference of the adjustable harness 1228. In oneembodiment, the adjustable straps 1230 are adjusted using a motorizedmechanism.

To perform the calibration of subjective force and objective force asdescribed above, a user wearing the adjustable harness 1228 mayindicate, for each level of adjustment of the adjustable strap 1230, alevel of subjective force due to the corresponding pressure sensor 1226pressing against the forearm surface 1232. For each level of subjectiveforce indicated, each pressure sensor 1226 indicates an objective levelof force by, for example, a voltage amount output by the pressure sensor1226 due to the force sensed by the pressure sensor 1226. Thisestablishes, for each pressure sensor 1226, and thus for each locationon the user's forearm corresponding to the location of the pressuresensor 1226, a relationship between objective force measured by thepressure sensor 1226 and the subjective level of force experienced bythe user. As shown further below with reference to FIGS. 12C through12F, such a relationship may be used to develop a calibration curve forthe user and to calibrate haptic devices to generate haptic output thatis calibrated in accordance with the user's sense of subjective force,such that all users experience a consistent level of subjective forcefor an intended level of haptic feedback.

In another embodiment, instead of using adjustable straps, an adjustableharness 1236 of the haptic device 1200 may include adjustable spacers1238 on the interior surface of the adjustable harness 1236, each havinga pressure sensor 1240 attached to the side of the adjustable spaceopposite the interior surface of the adjustable harness 1236 and facingthe forearm surface 1242. Each adjustable spacer 1238 may adjust itslength to increase or decrease the distance between the interior surfaceof the adjustable harness 1236 and the forearm surface 1242. This may beachieved via various mechanisms, such as a screw mechanism, inflatablebladder, spring mechanism, and so on. The adjustable spacer 1238 mayinclude markings to indicate the current level of adjustment.

The pressure sensor 1240, which may be similar to the pressure sensor1226, senses the pressure due to the combination of the adjustableharness 1236 and adjustable spacer 1238 on the forearm surface 1242. Theuser, as in the case of the adjustable harness 1228, may adjust thelength of the spacer and indicate a level of subjective force, and thesystem may determine the amount of actual force indicated by eachpressure sensor 1240 for the corresponding subjective force experienced.

In another embodiment, instead of using pressure sensors, such aspressure sensors 1226 and 1240, the adjustable harness includescutaneous actuators, such as cutaneous actuators 1104. In this case,instead of measuring force, the system determines the haptic outputexerted by the cutaneous actuator versus the subjective haptic outputexperienced by the user. Additional details regarding this alternativemethod are described below with reference to FIGS. 12D through 12F.

FIG. 12C is a block diagram illustrating the components of a system forcalibrating haptic feedback on a per-user basis using subjective forceinputs and voltage readings from sensors, according to an embodiment.These components may be stored as computer readable instructions inmemory, such as the memory 813. In another embodiment, these componentsare separately implemented using dedicated hardware, including aprocessor and memory.

The subjective magnitude input device 1208 receives inputs from a userindicating a subjective level of force at a particular location on theuser's body corresponding to a location where a pressure sensor, such aspressure sensor 1226, is located. The level of subjective force may bespecified on an arbitrary scale, such as from 0-10 (with 10 beinghighest). The lowest level of subjective force may be that where noforce is sensed by the user, or where any sensation of force is onlyperceived 50% of the time. The highest level may be one where anyadditional increase in force does not cause a change in perception bythe user, or where the subjective feeling of force becomes uncomfortableor painful. The location may be specified by some identifier of thepressure sensor, via interfacing a visual representation of the user'sbody (or portion of the user's body), and so on. The subjectivemagnitude input device 1208 may include a type of input device, such asa keypad, keyboard, voice recognition system, or other device, whichallows a user to input the subjective level of force values along withthe location or pressure sensor for which the subjective level of forceis indicated.

The sensor voltage input 1210 is a value indicating an objectivemeasurement of force sensed by the corresponding pressure sensor forwhich the user provides a subjective level of force via the subjectivemagnitude input device 1208. This objective measurement may be a voltageoutputted by the pressure sensor. It may also be measured in some otherformat, such as decibels, Newtons, and so on. This information may berequested from the corresponding pressure sensor when it is indicated bythe user via the subjective magnitude input device 1208.

The per-user force calibration data generator 1212 receives thesubjective level of force from the subjective magnitude input device1208 and the sensor voltage input 1210 for multiple pressure sensors andat multiple levels of subjective force, and generates a set ofcalibration data 1220 for the user undergoing the force calibrationprocess described herein. The per-user force calibration data generator1212 includes a force-voltage logger 1214, a force voltage database1216, a calibration data generator 1218, and the output calibration data1220. The calibration data 1220 may be used to calibrate a hapticdevice, such as haptic device 1224.

The force-voltage logger 1214 logs the input values of the subjectivelevel of force from the subjective magnitude input device 1208 and thecorresponding sensor voltage inputs 1210. This may occur each time avalue is entered into the subjective magnitude input device 1208. Theforce-voltage logger 1214 stores the associations between the receivedsubjective level of force and the sensor voltage input for thecorresponding pressure sensor to the force voltage database 1216.

The calibration data generator 1218 generates calibration data 1220 foreach location on the user's body for which a set of sensor voltageinputs and subjective level of force measurements are available. Thecalibration data 1220 indicates for each level of force thecorresponding voltage of a pressure sensor at that location, and isunique for each user. This calibration data 1220 may be represented in agraph, as described below with reference to FIG. 12D. This calibrationdata 1220 may be transmitted to a haptic signal generator 1222 ordirectly to a haptic device 1224 (as indicated by the dotted lines).When the user wears the haptic device 1224, the haptic signal generator1222 uses the calibration data 1220 when transmitting actuator signalsto the haptic device 1224. An example of how this calibration data 1220is used is described below.

An application, such as application 814 of FIG. 8, may request a hapticfeedback to be transmitted to the haptic device 1224. Regardless of theuser wearing the haptic device 1224, the application transmits the samehaptic feedback request. This would result in the haptic device 1224generating the same objective level of haptic output on the user but mayalso result in different subjective levels of force experienced for eachuser. Using the calibration data 1220, the haptic signal generator 1222is able to modify the actuator signals sent to the haptic device 1224such that this difference subjective experience does not occur, andinstead, every user experiences the same subjective level of hapticoutput. As the calibration data 1220 indicates a sensor voltage forevery level of subjective force, the haptic signal generator 1222 maymodify the actuator signal to request the haptic device 1224 to eithergenerate a stronger haptic output from a default value if it is the casethat the user's subjective level of haptic output is lower than abaseline, or to generate a lower haptic output from a default value inthe case where the user's subjective level of haptic output is greaterthan a baseline. In this case a baseline may be a hypothetical scenariowhere the subjective level of haptic output experienced by a user isexactly the same as a corresponding sensor voltage input, i.e., a one toone relationship between sensor voltage and subjective haptic output.This is unlikely to be the case as the subjective level of haptic outputis likely to be non-linear in relation to the sensor voltage input.However, this baseline may be used as the default case if calibrationdata 1220 were unavailable. Note that before applying the calibrationdata 1220, the calibration data 1220 may be normalized to match a scaleof the actuator signals for the haptic feedback device.

Although a haptic signal generator 1222 is described here as receivingthe calibration data 1220, the calibration data 1220 may in some casesbe received directly by the haptic device 1224, which modifies incomingactuator signals according to the method described above with referenceto FIG. 12C.

Although the above description is made in regards to pressure sensors,in other embodiments the sensors may not be pressure sensors, but arerather other sensors, such as temperature sensors, acceleration sensors,vibrational sensors, shear sensors, and so on. In yet other embodiments,the sensor voltage input 1210 is not from a sensor, but insteadindicates a value generated by a cutaneous actuator indicating a levelof activation for that cutaneous actuator. In one embodiment, the valueis a voltage indicating an amount of force applied by the cutaneousactuator. In these alternative embodiments, the subjective magnitudeinput device 1208 may also receive different types of subjectivemagnitude measurements instead of measurements of subjective forcelevels, with the type of subjective magnitude indicated by the usercorresponding to the type of haptic output generated by the cutaneousactuator. As an example, if the cutaneous actuator were outputting avibrational haptic output, the subjective magnitude indicated by theuser would be a magnitude of vibration.

In another embodiment, the subjective magnitude input device 1208 mayreceive the indication of the subjective magnitude levels while the useris performing different activities, such as running, walking, talking,working out, sitting, and so on. The subjective magnitude measuredduring these various events may differ from each other, and thusadditional calibration data 1220 may be generated for each of theseactivities. When the calibration data 1220 is used, different sub-setsof the calibration data 1220 may be utilized depending upon thecurrently detected activity.

FIG. 12D illustrates a graph representation of the calibration data 1220described with reference to FIG. 12C. Three different graphs for threedifferent sensors, such as pressure sensors 1226A-C, are shown. However,in other embodiments, there may be more sensors, and thus more sets ofcalibration data. In addition, in other embodiments, the sensors maymeasure other values aside from pressure, such as vibration, and thesubjective level of force measurement would match accordingly and be asubjective measurement of the user of the corresponding value.

As shown with the graph plotting the per-user subjective force 1274 withthe voltage 1276 for sensor 1, the subjective level of force measured bythe user is not co-linear with the voltage, but instead variesnon-uniformly. Note that the graphs shown here are examples only, and inpractice the non-uniformity may not be as pronounced as shown here.However, note that the subjective level of force measurement increasesmonotonically with the increase in voltage 1276.

Additional sets of calibration data 1220 are collected for additionalsensors. Thus, illustrated are graphs representing such calibration data1220 for a sensor 2 with a subjective level of force 1278 plottedagainst voltage 1280, and for a sensor N with a subjective level offorce 1282 plotted against a voltage 1284. The greater the number ofsensors, the more accurate the calibration data 1220 for a user can be,as the subjective levels of force for more locations on the user's bodywhere a haptic device will make contact are measured. When a hapticdevice applies a haptic output to these same locations, thecorresponding calibration data 1220 for that location may be used sothat an appropriate subjective level of force is sensed by the user, asdescribed previously.

Note that the graph for sensor N is represented by a straight line,indicating co-linearity between the subjective level of force 1282 andthe voltage 1284. This is unlikely to occur in practice, however, thisgraph also illustrates a baseline curve, which may be a reference fromwhich a haptic device may adjust the voltage to a cutaneous actuator inorder to calibrate its haptic output for a user. When an actuator signalis received by the haptic device, it may indicate a particular voltageto be applied to a cutaneous actuator. The haptic device may first findin the calibration data 1220 a sensor, which when the calibration data1220 was generated, was at the same location on the user's body as thelocation of the cutaneous actuator. The haptic device also determines anobjective level of haptic output (e.g., a force value measured inNewtons) for the cutaneous actuator for the voltage to be applied, andfinds the equivalent voltage for the sensor when the sensor sensed thesame level of haptic output. Such an equivalence between voltage andhaptic output levels may be stored in the calibration data 1220. Thehaptic device may determine in the calibration data 1220 for the sensorif the subjective level of force at that voltage level for the sensor isabove or below the baseline for that voltage level. As noted above, thisbaseline is represented by the graph for sensor N. If it is above thebaseline, then the haptic device may lower the voltage to be applied tothe cutaneous actuator such that the user would experience thesubjective level of force matching the baseline value. Conversely, if itbelow, then the haptic device may increase the voltage to be applied tothe cutaneous actuator. For example, at a voltage level of 1V for asensor, corresponding to an objective level of force of 1 N, thesubjective level of force may be measured as 5 in the baseline. However,for the user in question, a voltage level of 1V corresponds to asubjective level of force of 6 at the particular location on the user'sbody (assuming that the equivalence of voltages between the sensor andcutaneous actuator is one to one). This means that the user feels astronger subjective level of force than the baseline for the sameobjective force. Consequently, the haptic device, when applying a forceto the same location on the user's body, may lower the voltage such thatless than 1 N is applied, but resulting in a subjective level of forceexperienced that is in accordance with the baseline value, i.e. 5.

The change to the voltage may also be determined using the calibrationdata 1220. The haptic device may determine the point in the calibrationdata 1220 for the sensor where the user experienced a subjective levelof force that matches the baseline level of force (1 N in the aboveexample). The haptic device determines the voltage from the sensor atthat subjective level of force, and determines the equivalent voltagefor the cutaneous actuator. This is the modified voltage that is appliedto the cutaneous actuator. Following from the example above, thecalibration data 1220 indicates that at 0.8V for the sensor, the userexperienced a subjective level of force of 5. Again, assuming that theequivalence between the voltage of the sensor and the cutaneous actuatoris one to one, the haptic device would apply a voltage of 0.8V to thecutaneous actuator, instead of 1V, resulting in the desired subjectivelevel of force of 5 and matching the baseline.

FIG. 12E illustrates an alternative arrangement of generatingcalibration data using cutaneous actuators, according to an embodiment.In FIG. 12E, the adjustable harness uses cutaneous actuators, such asthose described above, in order to generate the calibration data 1220.This may be the same cutaneous actuator used to ultimately generate thehaptic output to the user. Two arrangements are shown in FIG. 12E. Inthe first one, cutaneous actuator 1256 is placed on the interior surfaceof an adjustable harness. A pressure sensor 1256 is placed on acutaneous actuator 1256 on a side of the cutaneous actuator 1256opposite the side facing the interior surface of the adjustable harness.The pressure sensor 1256 contacts the forearm surface 1264 of theforearm 1262 when the adjustable harness 1260 is worn by the user on theuser's forearm. Instead of using an adjustable strap or other adjustablemechanism to change the force applied by the adjustable harness 1260,instead the cutaneous actuator 1258 is activated at different levels ofhaptic output, with the pressure sensor 1256 recording a sensor voltagelevel at these different levels of haptic output. The user indicates asubjective level of force at the subjective magnitude input device 1208for each of the levels of haptic output, while the pressure sensor 1256provides the corresponding sensor voltage input.

In the second arrangement, no pressure sensor is present. Instead, acutaneous actuator 1266 is attached to the interior surface of theadjustable harness 1270 and makes contact directly with the forearmsurface 1268 of the forearm 1272. As with the adjustable harness 1260,no adjustment mechanism is used to modify a level of force. Instead, thecutaneous actuator 1266 directly provides a haptic output to the forearmsurface 1268. The input to the cutaneous actuator 1266, whether it be avoltage or other value, is provided as the sensor voltage input 1210,instead of a sensor voltage value. As before, the user provides thesubjective magnitude to the subjective magnitude input device 1208.Here, the cutaneous actuator 1266 may provide any type of hapticoutput—not just a pressure type. For example, the cutaneous actuator1266 may provide a vibration type output, a shear force output, adisplacement type output, an electrical output, a temperature output, anauditory output, and so on. These may all generate a haptic sensationwith the user, and the user may indicate the subjective magnitude of thesensation with the subjective magnitude input device 1208. This methodallows the system to calibrate for a user's sensation for any type ofcutaneous actuator. A representation of the calibration data 1220 isdescribed in detail below with reference to FIG. 12F.

FIG. 12F illustrates an example representation of calibration data as agraph using a cutaneous actuator as in FIG. 12E, according to anembodiment. In the illustrated graph, an input frequency 1288 into thecutaneous actuator, such as cutaneous actuator 1266 is plotted on thehorizontal axis, and a subjective magnitude 1286 of the haptic output isplotted on the vertical axis. The cutaneous actuator in this case mayprovide a vibrational haptic output. As the frequency of the vibrationincreases, as shown in the input frequency 1288 axis, the subjectivemagnitude 1286 that is perceived increases. In one embodiment, thesubjective magnitude may be measured as above, i.e., in accordance withthe user's perception of the strength of the haptic output. For example,in the illustrated example, the subjective magnitude is measured byfirst determining, at a given input frequency 1288 for the cutaneousactuator, a minimum amplitude of voltage to the cutaneous actuator wherea user may detect the haptic output (the vibration) at least 50% of thetime over multiple tests. At this level, the detection of the user ismeasured in decibel (dB) relative to a 1 micron peak or dB relative to1N displacement. In other words, the subjective magnitude is computed as20*log₁₀(x), where x is the objectively measured force or displacementcreated by the cutaneous actuator. Over the range of input frequencies1288, a human detection threshold curve is thus generated. The inverseof this curve is then used as the calibration data 1220 and is the oneshown in FIG. 12F. In other words, the subjective magnitude is theinverse of the detection threshold for the cutaneous actuator over therange of input frequencies.

In another embodiment, the subjective magnitude of the haptic output isadjusted in an array with a reference level. This generates aniso-sensitivity curve. Such an iso-sensitivity curve is determined atvarious reference levels of objective haptic output to generatecalibration data across all levels of haptic output. Eachiso-sensitivity curve is similar to that shown in FIG. 12F, however, thecalibration data includes multiple iso-sensitivity curves, with eachcorresponding to a different level above the minimum sensation level(e.g., 0 dB above, 10 dB above, 20 dB above, and so on). The minimumlevel is represented by the curve shown in FIG. 12F, and each of theiso-sensitivity curves with levels above the minimum level can berepresented as curves that are parallel to the curve shown in FIG. 12F.A user may adjust one or more cutaneous actuators to use one of theseiso-sensitivity curves. This increases or decreases the subjectivemagnitude experienced by the user, although other aspects, such as thefrequency of the haptic output from the cutaneous actuator, remains thesame.

FIG. 12G is a flowchart illustrating a method for gathering per-usercalibration data for haptic output, according to an embodiment. Theprocess described in the flowchart may be performed by the per-userforce calibration data generator 1212, which in turn may be a componentof a signal generator that may be part of a haptic device. Although aparticular arrangement of steps is shown here, in other embodiments theprocess may be arranged differently.

Initially, the per-user force calibration data generator 1212 receive1292 an indication of the subjective force value from a subjective forceinput device, such as the subjective force input device 1208. Theper-user force calibration data generator 1212 receives 1294 from atleast one pressure sensor, such as pressure sensor 1226, a sensorvoltage value.

The per-user force calibration data generator 1212 stores the subjectiveforce value and the corresponding sensor voltage value in a data store,such as the force voltage database 1216. Using this information, theper-user force calibration data generator 1212 generates 1298 acalibration curve indicating a correspondence between subjective forcevalues and sensor voltage values for each of the pressure sensors usingthe data from the data store. This calibration curve may be used tocalibrate a haptic device, such that each user experiences a samesubjective level of force for the same intended haptic output.

Neural Network Model for Generation of Compressed Haptic Actuator Signalfrom Audio Input

Embodiments also relate to using machine learning to generate a model toconvert an audio input to a sequence of haptic outputs of multiplecutaneous actuators in a haptic device for decoding by a human. While anautomated system may be able to encode audio into and decode audio fromhaptic output in an arbitrarily complex fashion, the requirements forhuman decoding of the haptic output are different. In particular, theencoding of audio signals for human-capable decoding should followvarious constraints, such as high compressibility and low entropy. Sucha conversion may be achieved by training a machine learning model toconvert the audio input to haptic outputs within these constraints.

FIG. 13A is a block diagram illustrating an unsupervised learning module1300 used to train a neural network 1310 in compressing an audio inputto a sequence of haptic cues, according to an embodiment. Theunsupervised learning module 1300 may be used to generate a set ofcoefficients and other settings for the neural network 1310. In oneembodiment, the neural network 1310 is also known as a machine learningcircuit or a machine learning algorithm. The components of the blockdiagram in FIG. 13A may be stored as computer readable instructions inmemory, such as the memory 813. In another embodiment, these componentsare separately implemented using dedicated hardware, including aprocessor and memory.

The unsupervised learning module 1300 receives an acoustic signal 1302as input. The input may be any audio waveform, and may be encoded in anyformat, at any bit rate, sample rate, etc.

The pre-processor 1304 of the unsupervised learning module 1300 firstconverts the acoustic signal 1302 into a spectrogram 1306. Such aspectrogram can be generated using a Fourier transform of the originalacoustic signal 1302. If the input acoustic signal 1302 is digital(i.e., is sampled), then a discrete Fourier transform may be used toconvert the acoustic signal 1302 to the spectrogram 1306. Thespectrogram 1306 represents the magnitude of various frequencies overwindows of time for the input acoustic signal 1302. For example, thespectrogram 1306 may represent the acoustic signal 1302 in 32 ms slices.The spectrogram 1306 represents the audio in each slice by frequency,and indicates for each frequency the magnitude of the frequency in thatslice. In one embodiment, the magnitude is represented by the log of theamplitude (i.e., the log amplitude). The magnitude and frequency valuescan be represented numerically, and these numerical representations areused as the input feature for the neural network 1310. In anotherembodiment, the spectrogram 1306 is a Mel-frequency cepstrum, and isrepresented using Mel-frequency cepstrum coefficients (MFCCs).

The neural network 1310 is a machine learning model that is trained inorder to receive as input the pre-processed acoustic signal 1302 andgenerate as output a sequence of actuator signals representing theacoustic signal 1302 and also obeying a variety of constraints 1312. Theneural network 1310 may be a convolutional neural network. Theconvolutional neural network processes on multiple slices (or frames) ofthe spectrogram in order to generate a sequence of actuator signals.These actuator signals are intended to be a representation of the centerslice among the multiple slices that are processed. In anotherembodiment, the neural network 1310 is a recurrent neural network, whichhas nodes that form a directed cycle. In this fashion, the neuralnetwork 1310 may process each slice of the spectrogram 1306 separately,but with each slice influencing the state of the neural network 1310 forthe processing of the subsequent slice. The neural network 1310 may useany one of a common set of activation functions, such as the sigmoid,softmax, rectifier, or hyperbolic tangent. The input features are fedinto the neural network 1310, which has been initialized with randomizedweights. The output of the neural network 1310 indicates a combinationof haptic cues for each slice of the input audio. Each of these hapticcues indicates the haptic output for a cutaneous actuator, such ascutaneous actuator 108, and may be represented by one or more nodes inthe final layer of the neural network 1310. Each node of the neuralnetwork 1310 may indicate for each cutaneous actuator 108, a percentagevalue that may be used to determine whether or not to activate aparticular state of the cutaneous actuator. For example, if thepercentage is below 50%, then the cutaneous actuator should not beactivated. Multiple nodes may be used if the cutaneous actuator has morethan two states.

The neural network 1310, during training, is also constrained based onthe various constraints 1312, which will be described in further detailbelow. These constraints 1312 aim to constrain the output of the neuralnetwork 1310 towards high compressibility, low entropy, low ordinality(i.e., fewer number of possible haptic outputs), sparsity, temporalstability, and spatial stability. Subsequent to the training process,the neural network 1310 is able to produce a set of haptic cues for aset of cutaneous actuators which represents the input acoustic signal1302. When the haptic outputs indicated by these haptic cues are sensedby a user, the user, who may be trained, may then be able to recreate,or at least have an understanding of the input acoustic signal. Thehaptic output, if recorded, may also be used to reconstruct the originalinput acoustic signal. In this fashion, an acoustic signal can becompressed into a haptic (or tactile) signal.

The cost analyzer 1316 determines, during the training of the neuralnetwork 1310, the cost of the current iteration of the neural network1310. The cost is computed based on the error between a reconstructedversion of the acoustic signal using the current output of the neuralnetwork 1310 in addition to how well the current iteration of the neuralnetwork 1310 meets the constraints 1312.

The reconstruction of the haptic cues into the original speech signalmay be achieved via a second neural network, hereinafter called areconstruction neural network. The reconstruction neural network istrained on the reverse of the actions of the neural network 1310. Itreceives as input the compressed haptic cues from the neural network1310 and is trained to attempt to generate an output that most closelymatches the original input features (the spectrogram 1306) with as smallof an error amount as possible. The training of the reconstructionneural network attempts to reduce this error amount (e.g., usinggradient descent) until a satisfactory minima is reached (e.g., after acertain number of iterations). At this point, the error between thereconstructed spectrogram generated by the reconstruction neural networkand the input spectrogram 1306 is compared and an error amount isdetermined. In one embodiment, the reconstructed spectrogram is furtherconverted into a reconstructed audio signal, which is compared with theinput audio signal 1302 to determine the error. The error may becomputed via cross power spectral density, or other methods. In otherembodiments, the error is not computed using a reconstruction neuralnetwork, but using some other function that attempts to determine arelationship between the output actuator signals and the spectrogram1306, such as a correlation. The difference in the identifiedrelationship may be deemed to be the error amount.

In addition to the error amount, the cost also includes a computationbased on how well the current iteration of the neural network 1310 meetsthe constraints 1312. These constraints 1312 may compute a cost valuebased on the output of the neural network 1310 and/or the hidden layerswithin the neural network 1310. Additional details regarding theindividual constraints 1312 are described immediately below.

The constraints 1312 are various restrictions on the output of theneural network 1310 and are implemented in the training of the neuralnetwork 1310 by modifying the cost function used to train the neuralnetwork 1310. A list of exemplary constraints 1312 and how they areachieved by modifying the cost function are described below. Two goalsof the constraints 1312 are to achieve a high compressibility—such thata high throughput is achieved with a limited number of haptic outputs,and low entropy—such that the haptic outputs are not highly disorderedbut rather have a discernable pattern. In one embodiment, theconstraints 1312 include 1) low ordinality, 2) high sparsity, 3) hightemporal stability, and 4) high spatial stability.

1) A first constraint 1312 is low ordinality. This constraint 1312 is tolimit the number of the types of haptic cues to a manageable amountwithin a human ability to detect the different types of output. Inparticular, the types may be limited to an off state, and two on states(represented at the actuator with a 1) 100 Hz sinusoid and a 2) 250 Hzsinusoid in one embodiment). These indicate the frequency at which acutaneous actuator vibrate at when on. In one embodiment, thisconstraint of low ordinality is achieved by indicating a lower cost foroutputs from the neural network 1310 that achieve the indicated numberof outputs, and indicating a very high cost for those that do not. Inanother embodiment, this constraint of low ordinality is achieved byhaving the neural network 1310 configured to output two nodes for eachcutaneous actuator, one node indicating a 100 Hz state, and the otherindicating at 250 Hz state. The cost function in this scenario would bemodified such that a high cost is given if any of these output nodes forone cutaneous actuator are simultaneously active.

2) A second constraint 1312 is high sparsity. This constrains the numberof cutaneous actuators that are activated at once as humans may becomeconfused when exposed to more than a few active cutaneous actuators.This sparsity constraint 1312 may be achieved by adding a cost that isproportional or increases with the number of cutaneous actuators thatare indicated to be active by the neural network 1310 for a set ofhaptic cue outputs at a time slice. For example, the cost function forsparsity may compute the number of nodes indicating activation ofcutaneous actuators, and derive a cost value in proportion to the numberof nodes. In another embodiment, this sparsity constraint 1312 isachieved by filtering out outputs from the neural network 1310 thatexceed a max number of activated cutaneous actuators. The result fromthe iteration of the neural network is discarded and a new iteration isperformed with newly modified weights.

3) A third constraint 1312 is temporal stability. This constrains thefrequency of changes in the state of cutaneous actuators so that they donot change too rapidly, which may cause confusion as a human may not beable to perceive such rapid changes. This temporal stability constraint1312 may be achieved by adding a cost that is proportional to the numberof changes from the output of the previous slice of the inputspectrogram 1306. As the neural network 1310 may be a recurrent neuralnetwork, it can recall information from a prior state, and thus the costmay be increased when the prior information indicates that a change hasoccurred.

4) A fourth constraint 1312 is spatial stability. This constraint thespatial variability of activated cutaneous actuators such that they arenearer together spatially on a haptic device. In other words, cutaneousactuators that are activated should be clustered close together. In oneembodiment, this constraint is achieved by having only certain groups ofcutaneous actuators to be on at any given time. Thus, if the neuralnetwork 1310 output indicates that cutaneous actuators from more thanone group are active at one time, the cost for this constraint may beindicated to be very large. Alternatively, the result may be dropped andthe neural network executed with newly updated weights. In anotherembodiment, the cost for this constraint may be achieved by decreasingthe cost for outputs from the neural network 1310 where adjacentcutaneous actuators are active at the same time while fewer cutaneousactuators which are further away are active. For example, for eachcutaneous actuator that is active, a base cost is indicated. However,for each adjacent cutaneous actuator that is active, the base cost issubtracted by an amount which may correspond to the distance to theadjacent cutaneous actuator. The sum of all the costs are added togetherto generate the total cost for the spatial stability constraint. Inaddition, the neural network 1310 may receive as input data regardingthe configuration and distances between cutaneous actuators. This mayprovide the additional information to the neural network 1310 that mayassist in optimizing training time.

Although four different constraints 1312 are described above, in otherembodiments, the constraints 1312 include more than these four. In onecase, an additional constraint is the inclusion of a joint phonemeclassifier. The joint phoneme classifier is trained upon the output ofthe neural network 1310, and the error rate from this classifier isincluded in the overall cost function computed by the cost analyzer1316. This may allow for an increase in the correspondence between thehaptic cues and any phonemes represented in the haptic cues.

The cost analyzer 1316 computes a weighted average of the costs for eachconstraint and the error from the reconstruction of the haptic cues(over ever time slice) to derive a final cost value. This is transmittedto the neural network updater 1314 which updates the neural network 1310based on this cost.

The neural network updater 1314 updates the neural network 1310 based onthe computed cost from the cost analyzer 1316. This may be achievedusing standard neural network training mechanisms, such asbackpropagation and stochastic gradient descent. The process of updatingthe neural network 1310 and analyzing the cost via the cost analyzer1316 are repeated until a desired error rate is achieved or whereaccuracy is no longer improved when testing against a validation set.Other standard indicators for stopping training may also be used.

FIG. 13B is a block diagram illustrating the use of the neural network1310 after it has been trained, according to an embodiment. An acousticsignal 1318 (not part of the training set used to train the neuralnetwork 1310) is received by the pre-processor 1304, which converts itinto a spectrogram 1306. The trained neural network 1310 receives thespectrogram 1306, and outputs a set of haptic cues 1322 for each timeslice of the spectrogram. These haptic cues 1322 indicate a pattern ofhaptic outputs for the cutaneous actuators of the haptic device 1324,but may not yet be in a format that can be used by the haptic device1324 directly. For example, the haptic cues 1322 may be output as a setof percentage values, as described above with reference to FIG. 13A.Thus, the haptic cues 1322 may first be converted into actuator signalsby a haptic signal generator 1323, before being transmitted to thecutaneous actuators 1324 to generate the corresponding haptic outputs.

FIG. 13C illustrates a conversion using the neural network 1310 of aspeech signal into a set of haptic outputs for a set of cutaneousactuators 1330, according to an embodiment. The speech signal 1326 maybe a sampled waveform, such as the one illustrated. This speech signal1326 is converted 1328 using the process described above into a sequenceof haptic cues, which are subsequently transmitted to a haptic deviceand causes a set of cutaneous actuators 1330A-N on the haptic device togenerate a set of haptic outputs, as shown. After sensing these hapticoutputs, a human may be able to determine information about the originalspeech signal 1326. This may allow a human to understand some aspects ofthe original speech signal 1326 without having hear the speech signal1326 or via any other auditory means.

FIG. 13D is a flowchart illustrating a method for training a machinelearning circuit to generate a set of compressed haptic cues from anacoustic signal, according to an embodiment. The process described inthe flowchart may be performed by the unsupervised learning module 1300.Although a particular arrangement of steps is shown here, in otherembodiments the process may be arranged differently.

The unsupervised learning module 1300 receives 1336 a training set ofspeech signals, such as the speech signal 1302. The unsupervisedlearning module 1300 pre-processes 1338 the training set of speechsignal into pre-processed audio data. The pre-processed audio data mayinclude at least a spectrogram, such as the spectrogram 1306. Theunsupervised learning module 1300 trains 1340 a neural network using thepre-processed audio data. Additionally, the neural network generates1342 a sequence of haptic cues corresponding to the speech signal. Theneural network has a cost function based on a reconstruction error and aplurality of constraints, such as the constraints 1312. Theseconstraints cause the sequence of haptic cues that are output to havehigh compressibility and low entropy. The sequence of haptic cues aretransmitted cutaneous actuators to generate a sequence of hapticoutputs. These cutaneous actuators face a skin surface, such as forearmsurface, of a user's body.

Speech Algorithm to Apply Haptic Communication Patterns Related toConsonant-Vowel Pairs or Syllables

Embodiments also relate to a haptic symbol set which specify sequencesof actuator signals for operating haptic actuators to be generated frominput words of a language. However, unlike the system described withregards to FIGS. 11A through 11J, the haptic symbols here are based onunits within the written form of the language, rather than phonemes,which are represented in the spoken component of the language. Suchunits may include consonant-vowel pairs or syllables. Unlike a languagesuch as English, which has a written system that does not break downinto syllables or consonant-vowel pairs, other writing systems of otherlanguages have such a feature. For example, the hiragana writing systemof Japanese is a syllabary, while the Devanagari script of Hindi is anabugida, i.e., it uses consonant-vowel pairs in the writing system.

FIG. 14A is a block diagram illustrating the components of a system forconverting the syllables of input words 1404 to actuator signals 1412 toactivate cutaneous actuators 1414 of a haptic device, according to anembodiment.

The input words 1404 are words in a language that can be formed from alimited number of syllables. While languages such as English (and otherEuropean languages), when spoken, are formed as syllables as well, thenumber of possible syllables in such a language exceed 10,000, and thusthe syllables of such a language cannot be easily represented by uniquesequences of actuator signals. Instead, the input words 1404 may be froma language that can be formed using a limited number of syllables (e.g.,under 100), such as with Japanese.

The syllable-haptic signal converter 1402 is similar to the word-hapticsignal converter 1107, but converts the input words 1404 into componentsyllables, and converts these syllables into sequences of actuatorsignals, instead of converting into phonemes. The syllable-haptic signalconverter 1402 includes a syllabic splitter 1406, a syllable-hapticsignal converter 1408 and a syllable-haptic signal database 1410. In oneembodiment, the syllable-haptic signal converter 1402 may be a componentin a signal generator, such as signal generator 800.

The syllabic splitter 1406 converts the input words 1404 into componentsyllables. The output of the syllabic splitter 1406 may be a set ofsymbols representing the syllables of the language of the input words1404. For example, Japanese may be represented by 46 syllables, asindicated by the hiragana writing system. The syllabic splitter 1406 mayconvert words to syllables based on a syllable database which indicatesthe syllables that correspond to the letters or other elements in thewritten form of the input words 1404.

The syllable-haptic signal converter 1408 may be similar to thephoneme-haptic signal converter 1109, but converts the syllables of theinput words 1404 into sequences of actuator signals, instead of thephonemes of the input words. For each syllable received from thesyllabic splitter 1406, the syllable-haptic signal converter 1408accesses the syllable-haptic signal database 1410 to determine thecorresponding sequence of actuator signals 1412 to output. In oneembodiment, each syllable has a single corresponding sequence ofactuator signals 1412.

The sequence of actuator signals 1412 that are generated are not justpatterned based on which cutaneous actuators 1414 of a haptic device toactivate in sequence. Instead, the sequence of actuator signals 1412 mayalso be patterned such that each actuator signal 1412 in the sequencemay have a different duration, and such that the absence of a signal,combined with other actuator signals 1412, may also indicate aparticular sequence that corresponds to a syllable. Thus, the durationof signals, and the lack of signals, may all contribute to differentsequences of actuator signals 1412 that represent different syllables.Additionally, the actuator signals 1412 may not simply be binary (e.g.,on and off) but may be variable instead, allowing the haptic output ofthe cutaneous actuators to vary in intensity, and allowing hapticoutputs that follow complex patterns and waveforms.

The actuator signals 1412 output by the syllable-haptic signal converter1402 are transmitted to the cutaneous actuators 1414, which may besimilar to the cutaneous actuators 1104 of FIG. 11B. Each actuatorsignal 1412 activates one of the cutaneous actuators 1414 for a durationof time (and in some embodiments, for a particular intensity, frequency,and so on). At the point when the cutaneous actuator 1414 receives itsactuator signal 1412, it activates. When the same actuator signal 1412ceases, the cutaneous actuator 1414 deactivates. Alternatively, theactuator signals 1412 are digital in that they transmit a specifiedduration and start time (and other factors, such as intensity) for aspecified cutaneous actuator 1414. This digital signal is converted by aprocessor by the cutaneous actuators 1414 themselves, and subsequentlythe corresponding cutaneous actuator 1414 activates according to thespecified duration at the specified start time. A sequence of theseactuator signals 1412 can generate a unique pattern that can berecognized and distinguished by a user from other sequences of actuatorsignals 1412.

A set of examples of particular haptic outputs generated by the actuatorsignals 1412 corresponding to syllables is described below withreference to FIG. 14B.

FIG. 14B illustrates exemplary sequence of haptic syllable hapticoutputs for an example input word 1404, according to an embodiment. Inthe illustrated example, the input word 1404 is “Tōkyō,” or “Tokyo” inEnglish. This word includes 5 different syllables: 1) /to/, 2) /

/ or “u”, 3) /ki/, 4) /jo/ or “yo”, and 5) /

/ or “u”. Note that the second, fourth, and fifth syllables are vowels.The syllable-haptic signal converter 1402 converts each of thesesyllables into a sequence of actuator signals 1412, which aretransmitted to the cutaneous actuators 1414, which generate a hapticoutput based on the actuator signals 1414 and which are sensed by theuser. For ease of discussion, each of the sequence of haptic outputsassociated with a syllable is referred to as a haptic syllable. Thus,the five syllables here are translated into five haptic syllables, whichare sensed by the user. After training, the user may be able to identifythe word “Tokyo” after sensing these five haptic syllables. A similarprocess would occur for other words.

In the illustrated example of FIG. 14B, a 3×2 two dimensional array ofcutaneous actuators 1440-1450 is represented by concentric circles, witheach set of two concentric circles representing one cutaneous actuator.The cutaneous actuators 1440-1450 are labeled for haptic syllable 1430,but are not repeatedly labeled for haptic syllables 1432-1438 forclarity. However, the cutaneous actuators indicated in haptic syllables1432-1438 are the same as those in the same positions in haptic syllable1430. This two dimensional array of cutaneous actuators 1440-1450 maycorrespond to the two dimensional array of cutaneous actuators 1104 inFIG. 11A.

Each haptic syllable 1432 through 1438 indicates a sequence of hapticoutputs by this array of cutaneous actuators that corresponds to asyllable of the input word “Tokyo”. Each single haptic output in asequence of haptic outputs corresponds to activation of a cutaneousactuator, and is represented by a set of two solid concentric circlesfor that cutaneous actuator. In contrast, non-activated cutaneousactuators are represented by dotted concentric circles. Furthermore, atwo point sequence of haptic outputs is represented by two sets of solidconcentric circles, with an arrow in between the two sets indicating anactivation of a first cutaneous activator which is located in adirection immediately opposite the direction of the arrow, followed bythe activation of a second cutaneous actuator which is indicated in adirection immediately following the direction of the arrow. The twopoint sequence of haptic outputs may be similar to the sequencedescribed with reference to FIG. 11D. For example, for haptic syllable1432 for the syllable “/

/” is indicated by the activation of the cutaneous actuator 1442followed by the activation of the cutaneous actuator 1446, after a delayperiod, such as the SOA delay described above with reference to FIG.11D.

The sequences of haptic outputs for each haptic syllable may beseparated from each other by a delay period with no haptic output (e.g.,of 200 ms), or by a special delimiter haptic output (e.g., all actuatorsactivate at the same time for an interval of time).

In the illustrated example, 1) “to” corresponds to the haptic syllable1430 and activation of the cutaneous actuator 1450, 2) “u” correspondsto the haptic syllable 1432 and activation of the cutaneous actuator1442 followed by 1446, 3) “ki” corresponds to the haptic syllable 1134and activation of the cutaneous actuator 1444, 4) “yo” corresponds tothe haptic syllable 1436 and activation of the cutaneous actuator 1440followed by 1442, and 5) “u” corresponds to the haptic syllable 1438 andactivation of the cutaneous actuator 1442 followed by 1446 (same as 2).

The correspondence between each haptic syllable and each spoken syllablemay be selected based on articulation association, in a fashion similarto that described above for FIG. 11H. Furthermore, in one embodiment,the duration of haptic output for consonant syllables may be shorterthan that of vowel syllables, in a fashion similar to that describedabove for FIG. 11H. Although a particular combination of haptic outputsare shown here for each syllable, in other embodiments the combinationsof haptic outputs to syllables may differ.

FIG. 14C is a block diagram illustrating the components of aconsonant-vowel pair (Abugida) haptic signal converter 1416 forconverting consonant-vowel pairs of input words 1418 to actuator signals1426 to activate cutaneous actuators 1428 of a haptic device, accordingto an embodiment. These components may be stored as computer readableinstructions in memory, such as the memory 813. In another embodiment,these components are separately implemented using dedicated hardware,including a processor and memory. While FIG. 14A described a system forconverting syllables to actuator signals, FIG. 14C describes a systemfor converting consonant-vowel pairs to actuator signals. Theconsonant-vowel pair (Abugida) haptic signal converter 1416 may include,among other components, a consonant-vowel pair splitter 1420, aconsonant-vowel haptic converter 1422 and a consonant vowel hapticsignal database 1424.

The input words 1418 are words in a language that can be formed from alimited number of consonant-vowel pairs. An example of such a languageis Hindi, which uses the Devanagari script, and which represents thespoken language in consonant-vowel pairs. These types of languages arecalled abugida. Note that languages such as English, while representablein consonant-vowel pairs, would have a very high number of such pairs,and thus such a representation may not be feasible for conversion intosequences of actuator signals.

The consonant-vowel (C-V) pair haptic signal converter 1416 is similarto the syllable-haptic signal converter 1402, but converts the inputwords 1418 into component C-V pairs, and converts these C-V pairs intosequences of actuator signals, instead of converting into syllables. TheC-V pair haptic signal converter 1416 includes a consonant-vowel pairsplitter 1420, a consonant-vowel haptic signal converter 1422 and aconsonant-vowel haptic signal database 1424. In one embodiment, the C-Vpair haptic signal converter 1416 may be a component in a signalgenerator, such as signal generator 800.

The C-V pair splitter 1420 converts the input words 1418 into componentC-V pairs. The output of the syllabic splitter 1406 may be a set ofsymbols representing the C-V pairs of the language of the input words1418.

The C-V haptic signal converter 1422 may be similar to thesyllable-haptic signal converter 1408, but converts the C-V pairs of theinput words 1404 into sequences of actuator signals, instead of thesyllables of the input words. For each C-V pair received from the C-Vpair splitter 1420, the C-V haptic signal converter 1422 accesses theC-V haptic signal database 1424 to determine the corresponding sequenceof actuator signals 1426 to output.

The actuator signals 1426 output by the C-V pair haptic signal converter1416 are transmitted to the cutaneous actuators 1428, which may besimilar to the cutaneous actuators 1104 of FIG. 11B. Each actuatorsignal 1426 activates one of the cutaneous actuators 1428 for a durationof time (and in some embodiments, for a particular intensity, frequency,and so on). A set of examples of particular haptic outputs generated bythe actuator signals 1428 corresponding to C-V pairs is described belowwith reference to FIG. 14D.

FIG. 14D illustrates an example sequence of haptic C-V pair hapticoutputs for an example input word 1418, according to an embodiment. Inthe illustrated example, the input word 1418 is “India,” which iswritten as “

” in Devanagari script. This word includes three different C-V pairs: 1)i+

, 2) /

/+i, and 3) /j/+a. These are approximately pronounced as “in,” “di,” and“ya,” respectively. The C-V haptic signal converter 1416 converts eachof these C-V pairs into a sequence of actuator signals 1426, which aretransmitted to the cutaneous actuators 1428, which generate sequences ofhaptic output. For ease of discussion, each of the sequence of hapticoutputs associated with a C-V pair is referred to as a haptic C-V pair.Thus, the three C-V pairs here are translated into three haptic C-Vpairs. A similar process would occur for other words.

In the illustrated example of FIG. 14D, a 3×2 two dimensional array ofcutaneous actuators 1458-1468 is represented by concentric circles, witheach set of two concentric circles representing one cutaneous actuator.The cutaneous actuators 1458-1468 are labeled for haptic C-V pair 1452,but are not repeatedly labeled for haptic C-V pairs 1454 and 1456 forclarity. This two dimensional array of cutaneous actuators 1458-1468 maycorrespond to the two dimensional array of cutaneous actuators 1104 inFIG. 11A.

Each haptic C-V pair 1452-1456 indicates a sequence of haptic outputs bythis array of cutaneous actuators that corresponds to a C-V pair of theinput word “India,” similar to the process noted above for the hapticsyllables of FIG. 14B.

In the illustrated example, 1) “i+

” corresponds to the haptic C-V pair 1452 and activation of thecutaneous actuator 1462 followed by the cutaneous actuator 1464, 2) “/

/+i” corresponds to the haptic C-V pair 1454 and activation of thecutaneous actuator 1464, and 3) “/j/+a” corresponds to the haptic C-Vpair 1156 and activation of the cutaneous actuator 1458 followed by1460.

The correspondence between each haptic C-V pair and each spoken C-V pairmay be selected based on articulation association, in a fashion similarto that described above for FIG. 11H.

FIG. 14E is a flowchart illustrating a method of converting input wordsinto haptic output based on C-V pairs, according to an embodiment. Theprocess described in the flowchart may be performed by the C-V pairhaptic signal converter 1416, which in turn may be a component of asignal generator that may be part of a haptic device. Although aparticular arrangement of steps is shown here, in other embodiments theprocess may be arranged differently.

Initially, the C-V pair haptic signal converter 1416 receives 1472 aninput word which is a unit of a language that is written using C-Vpairs, such as Hindi. The w C-V pair haptic signal converter 1416converts 1474 the input word into its component C-V pairs.

The C-V pair haptic signal converter 1416 converts 1152 the C-V pairsinto a sequence of actuator signals. This sequence is formed from aconcatenation of sub-sequences of actuator signals. Each C-V pair in thelanguage corresponds to one of these sub-sequences of actuator signals.Each of the actuator signals are mapped to a cutaneous actuator of a twodimensional array of cutaneous actuators placed on an interior surfaceof a harness shaped to wrap around a user's arm. The interior surface ofthe harness facing the user's arm when worn by the user.

The C-V pair haptic signal converter 1416 transmits the sequence ofactuator signals to the two dimensional array of cutaneous actuators.This causes the cutaneous actuators to activate in accordance with thesequence of actuator signals, and in turn, causes the user to sense aseries of haptic outputs from the cutaneous actuators. If trained, theuser may then be able to recognize the haptic outputs as C-V pairs of aword, and decipher the input word from the haptic outputs. In thisfashion, a user may be able to recognize words haptically rather thanaurally or visually. A similar flow may be used for converting inputwords to syllables, and syllables to sequences of haptic output, andthus is omitted here for sake of brevity.

Haptic Communication System Using Cutaneous Actuators for Simulation ofContinuous Human Touch

Embodiments also relate to a haptic communication device including anarray of cutaneous actuators to generate haptic sensations correspondingto actuator signals received by the array. The array includes a firstcutaneous actuator to generate first haptic sensation at a firstlocation on a body of a user at a first time, and a second cutaneousactuator to generate second haptic sensation at a second location on thebody of the user at a second time later than the first time. The arrayof cutaneous actuators may be mounted on a substrate placed on the bodyof the user. A signal generator generates the actuator signals to causethe array to create continuous tactile motion along the body of the userfrom the first location to the second location. The signals correspondto words of a social touch lexicon.

FIG. 15A is a planar view of an example array 1502 of cutaneousactuators mounted on a substrate 1504, in accordance with an embodiment.The substrate 1504 may be placed or wrapped around a part of a user'sbody 1500, e.g., on a forearm, chest, back, thigh, neck, etc. Thesubstrate 1504 may be made of a flexible material such as plastics(e.g., polyethylene and polypropylene), rubbers, nylon, synthetics,polymers, etc. In one embodiment, the array 1502 of cutaneous actuatorsmay be mounted on a flexible substrate 1504 such as a soccer shin guard.The array 1502 of cutaneous actuators provides haptic sensations tocreate continuous tactile motion along the body 1500 of the user fromthe beginning of the array 1502 to the end of the array 1502. When theindividual cutaneous actuators are stimulated in a sequence such thatthe generating of the haptic sensations by each cutaneous actuatoroverlaps the generating of the haptic sensations by the next cutaneousactuator in the sequence, the array 1502 creates a sense of continuoustactile motion (e.g., stroking or moving touch) on the body 1500 ratherthan discrete actuation points, as described below with reference toFIG. 15B.

The array 1502 of cutaneous actuators generates haptic sensationscorresponding to actuator signals (described above with respect to FIG.2 and below with respect to FIG. 15B) received by the array 1502. Forexample, the actuator signals may correspond to words of a social touchlexicon to provide pleasant or calming touches to the body 1500 of theuser. The haptic sensations include sensations provided by one or moreof the cutaneous actuators mounted on the substrate 1504.

FIG. 15A illustrates an array 1502 of twelve cutaneous actuatorsarranged in two rows and six columns. The twelve points of vibrationsgenerated by the cutaneous actuators may be used to create hapticillusions on the body 1500. In one embodiment, high bandwidth, lightweight, portable and wearable cutaneous actuators may be used. Forexample, each cutaneous actuator may be a voice coil, a linear resonantactuator (LRA), an eccentric rotating mass (ERM) actuator, apiezoelectric actuator, an electroactive polymer actuator, a shapememory alloy actuator, a pneumatic actuator, a microfluidic actuator,some other type of transducer, or a combination thereof, as describedabove with reference to FIG. 1. For example, six voice coils may be usedto create tactile illusions by discrete placement and continuousstrokes. A psychophysical control model may be determined to specify howto parametrically control the cutaneous actuators, as illustrated anddescribed in detail below with reference to FIG. 15E.

Each cutaneous actuator is a transducer that converts an actuator signalin a form of electrical energy (e.g., voltage or current signals) tohaptic sensations at a particular location on the body 1500. The array1502 of cutaneous actuators includes a first cutaneous actuator 1508A tobegin generating a first haptic sensation at a first location on thebody 1500 of the user at a first time t1, as described and illustratedbelow with respect to FIG. 15B. The array 1502 includes a secondcutaneous actuator 1508B to begin generating a second haptic sensationat a second location on the body 1500 of the user at a second time t2later than the first time. The first cutaneous actuator 1508A and thesecond cutaneous actuator 1508B are separated by a spacing distance1512A along the body 1500 of the user. The spacing distance betweenother consecutive pairs of the cutaneous actuators may be the same ordifferent. In one embodiment, the spacing distance 1512A separating thefirst cutaneous actuator 1508A and the second cutaneous actuator 1508Bis less than 40 mm. For example, the cutaneous actuators may be placed35 mm apart along a line of the dorsal forearm 1500.

In one embodiment, the distance 1516 separating the first cutaneousactuator and a last cutaneous actuator of the array 1502 is less than 15cm. The distance 1516 between the first cutaneous actuator 1508A and thelast cutaneous actuator on the array 1502 is the length of the device1516 and corresponds to the length of the surface on the body 1500 thatexperiences the calming or pleasant touches. For example, the length1516 may be 14 cm along a line of the dorsal forearm 1500. Larger orsmaller spacing distances 1512A or lengths 1516 may be used to vary thesensations provided to the user. Different distances 1512A and lengths1516 may also be used to determine a preferred distance 1512A or length1516, as described below with respect to FIG. 15E.

The array 1502 of the cutaneous actuators may be a one-dimensionalarray, a two-dimensional array or a two-dimensional pattern havingcutaneous actuators spaced unevenly and distributed across the substrate1504. The array 1502 of the cutaneous actuators may include one or morerows of cutaneous actuators and one or more columns of cutaneousactuators. FIG. 15A illustrates an embodiment having a two-dimensionalarray. Cutaneous actuators 1508A, 1508B, and 1508C are located in afirst row of the array 1502 of the cutaneous actuators. Cutaneousactuator 1508D is located in a second row of the array 1502 of thecutaneous actuators. The spacing distance 1512B separating the first rowof cutaneous actuators (1508C) and the second row of cutaneous actuators(1508D) is less than 40 mm. For example, the rows may be placed 35 mmapart along a line of the dorsal forearm 1500.

Mathematical functions (psychophysical models) may be determined, asdescribed below with respect to FIG. 15E, that associate the illusorycontinuous movements produced by the array 1502 as a function ofstimulation time, frequency, amplitude of the vibrations, etc. Discretegrids of vibrating actuators (such as motors in a cellphone orsmartwatch) may thus be placed on the forearm (or along the skin) toevoke continuous movements such as when a finger or a hand is touchedand moved across the forearm (or the skin).

FIG. 15B is an illustration of example haptic sensations generated bythe first cutaneous actuator 1508A and the second cutaneous actuator1508B at different times, in accordance with an embodiment. The hapticsensations generated by the other cutaneous actuators mounted on thesubstrate 1504 are also similarly staggered in time and transmitted toother locations on the body 1500 of the user.

The haptic sensations may correspond to words of a social touch lexiconincluding words for greeting, parting, giving attention, helping,consoling, calming, pleasant, and reassuring touches. The hapticsensations may also correspond to spoken words from a user that areconverted to haptic output, as illustrated and described in detail belowwith respect to FIGS. 17A-17E. The different haptic sensations, whenstaggered in time and distance, create continuous tactile motion alongthe body 1500 of the user from the first location (location of the firstcutaneous actuator 1508A) to the last location (location of the lastcutaneous actuator). The continuous tactile motion is intended totransmit the words to the body 1500 of the user, as described above withrespect to FIGS. 2, 3, and 8A through 8F.

As illustrated in FIG. 15B, the first cutaneous actuator 1508A beginsgenerating the first haptic sensation at a first location on the body1500 of the user at a first time t1. In one embodiment, the hapticsensations may be haptic vibrations. The amplitude of the first hapticsensation enables the device to transmit sophisticated hapticcommunication effects. The information related to the words of thesocial touch lexicon described above are communicated by altering theamplitude (e.g., 1532) of the haptic sensations (e.g., vibrations). Bychanging the amplitude and patterns of vibration, a large number ofcombinations, rhythms or messages may be reproduced.

The second cutaneous actuator 1508B begins generating the second hapticsensation at the second location on the body 1500 of the user at asecond time t2. The second time t2 is later than the first time t1 tostagger the haptic sensations in time to produce the continuous tactiletouch motions. The first cutaneous actuator 1508A ends generating thefirst haptic sensation at a third time t3 after the second time t2. Theoverlap time 1536 between the beginning of the generating of the secondhaptic sensations by the second cutaneous actuator 1508B and the end ofthe generating of the first haptic sensation by the first cutaneousactuator 1508A provides the continuous tactile touch motions instead ofthe user experiencing isolated sensations at different locations on theuser's body 1500.

The generating of the first haptic sensation overlaps with thegenerating of the second haptic sensation for a time interval indicatingan overlap time 1536 of the haptic sensation. The second cutaneousactuator 1508B may transmit the second haptic sensation by altering thistime interval (t3−t2) between the third time t3 and the second time t2.In one embodiment, when LRAs are used as the cutaneous actuators, theperformance characteristics of the cutaneous actuators may allow forshorter start-stop times. Therefore, the times t1, t2 and t3 may bereadily increased or decreased to determine a preferred overlap timethat provides better communication of the haptic sensations and words ofthe social touch lexicon. Methods to determine a preferred overlap timeare described in greater detail below with reference to FIG. 15E.

The first cutaneous actuator 1508A may end generating the first hapticsensation (at time t3) after a time interval indicating a duration timeof the first haptic sensations. For example, the first cutaneousactuator 1508A may alter a time interval between the first time t1 andthe third time t3. The time interval between the first time t1 and thethird time t3 indicates the duration time t3−t1 of the first hapticsensation. The smooth continuous motions may be modeled as a function ofthe duration time t3−t1 of the haptic sensations for stroking motion.The duration time t3−t1 determines the speed of the stroke. The durationtime t3−t1 of the first haptic sensation may be increased or decreasedto determine which preferred duration time t3−t1 provides bettercommunication of the haptic sensations and words of the social touchlexicon.

In one embodiment, the second cutaneous actuator 1508B may be turned onafter the third time t3 has passed, in which case a pause may occurbetween speakers. Such a configuration may provide other types of hapticcommunication to the user's body 1500. Again, methods to determine apreferred pause time between the third time t3 and the second time t2may be implemented similar to the methods described below with referenceto FIG. 15E.

FIG. 15C is a flowchart illustrating an example process of generatinghaptic sensations to create continuous tactile motion along the body1500 of a user, in accordance with an embodiment. In some embodiments,the process may have different and/or additional steps than thosedescribed in conjunction with FIG. 15C. Steps of the process may beperformed in different orders than the order described in conjunctionwith FIG. 15C. Some steps may be executed in parallel. Alternatively,some of the steps may be executed in parallel and some steps executedsequentially. Alternatively, some steps may execute in a pipelinedfashion such that execution of a step is started before the execution ofa previous step.

An array 1502 of cutaneous actuators mounted on a substrate receives1540A actuator signals corresponding to words of a social touch lexiconincluding greeting, parting, giving attention, helping, consoling,calming, pleasant, and reassuring touches. A signal generator coupled tothe array 1502 may generate and transmit the actuator signals to thearray 1502 to cause the array 1502 to create continuous tactile motionalong a body 1500 of the user from a first location to the a location.

A first cutaneous actuator of the array of cutaneous actuators generates1540B a first haptic sensation corresponding to the actuator signals.The first cutaneous actuator of the array of cutaneous actuatorstransmits 1540C the first haptic sensation to a first location on thebody 1500 of the user. The generating of the first haptic sensationbegins at a first time t1.

A second cutaneous actuator generates 1540D a second haptic sensationcorresponding to the actuator signals. In one embodiment, differenttypes of cutaneous actuators may be used to provide haptic illusionsthat vary in effect with the type and characteristics of the cutaneousactuators used. For example, LRAs may be used to provide shorterstart-stop times to alter the haptic illusions perceived by the user,while microfluidic actuators may be used to create a variety ofsensations including texture, size, shape, movement, etc.

The second cutaneous actuator transmits 1540E the second hapticsensation to a second location on the body 1500 of the user to createcontinuous tactile motion along the body 1500 of the user from the firstlocation to the second location. The continuous tactile motion isintended to transmit the words of the social touch lexicon to the body1500 of the user. The generating of the second haptic sensation beginsat a second time t2, which is later than the first time t1. Thegenerating of the first haptic sensation may end at a third time t3after the second time t2. Therefore, the generating of the first hapticsensation overlaps with the generating of the second haptic sensationfor a time interval indicating an overlap time of the haptic sensations.

FIG. 15D is a graph illustrating example Pacinian and Non Pacinianstimulation, in accordance with an embodiment. In one embodiment, thehaptic sensations may be haptic vibrations. The array 1502 of cutaneousactuators may generate the haptic sensations by altering an amplitude ora frequency of the haptic vibrations. FIG. 15D illustrates differenttypes of haptic stimulation that may be transmitted to the body 1500 ofa user by varying the amplitude and frequency of haptic vibrations.

The curve 1552A illustrates how Non Pacinian stimulation in the body1500 may vary with the amplitude of sensations and the frequency ofsensations. Pacinian and Non Pacinian stimulation in the body 1500 iscaused by mechanoreceptors, which are sensory receptors that respond tohaptic sensations, mechanical pressure, or distortion. Non-Pacinianmechanoreceptors (e.g., tactile corpuscles or Meissner's corpuscles) area type of nerve ending in the skin that is responsible for sensitivityto light touch. In particular, they are sensitive when sensing hapticsensations between 10 and 50 Hertz. The curve 1552A illustrates thatmore Non Pacinian stimulation is achieved when the frequency ofvibration is closer to 20 Hz. Less Non Pacinian stimulation is achievedwhen the frequency of vibration is closer to 70 Hz. Little Non Pacinianstimulation is achieved when the frequency of vibration is closer to 250Hz.

The Non Pacinian stimulation region 1556A (depicted in FIG. 15D)generally lies below a 70 Hz frequency of haptic sensations, i.e., thebody 1500 of the user may experience Non-Pacinian haptic stimulationwhen the cutaneous actuators produce haptic sensations below the 70 Hzfrequency. For example, a preferred frequency region of operation may bewhen the cutaneous actuators 1508 produce haptic sensations above 20 Hzfrequency and below 70 Hz frequency.

Pacinian mechanoreceptors (e.g., Lamellar corpuscles or Paciniancorpuscles) are nerve endings in the skin responsible for sensitivity tovibration and pressure. They respond to sudden disturbances and areespecially sensitive to sensations. For example, Pacinian stimulationmay be used to detect surface texture, e.g., rough vs. smooth. The curve1552B illustrates that more Pacinian stimulation is achieved when thefrequency of vibration is closer to 250 Hz. Less Pacinian stimulation isachieved when the frequency of vibration is closer to 70 Hz. LittlePacinian stimulation is achieved when the frequency of vibration iscloser to 20 Hz.

The Pacinian stimulation region 1556B generally lies above the 70 Hzfrequency of haptic sensations, i.e., the body 1500 of the user mayexperience Pacinian haptic stimulation when the cutaneous actuatorsproduce haptic sensations above the 70 Hz frequency. For example, apreferred frequency region of operation for Pacinian stimulation may bewhen the cutaneous actuators 1508 produce haptic sensations above 70 Hzfrequency and below 250 Hz frequency.

FIG. 15E is a graphical illustration of an example preferred region ofoperation 1568 for an array 1502 of cutaneous actuators, in accordancewith an embodiment. Using the methods described below with respect toFIG. 15E, preferred parameters of the haptic sensations (e.g.,frequency, amplitude, overlap time 1536, duration of sensations, etc.)and preferred parameters of the array 1502 (e.g., spacing distance 1512Aand 1512B, length of the device 1516, etc.) may be determined that aremore pleasant to human users. The preferred parameters may be used toassociate social touch messages such as calming and consoling.

The parameters of the haptic sensations may be examined across a broadspectrum of frequency and amplitude. In some embodiments, frequencies ofvibration in the 5-40 Hz range and amplitudes of up to 20 times that ofthe human sensitivity threshold evoke more pleasant touch. Theseparameters may be combined with the psychophysical models of movingillusions, social and consoling patterns to generate haptic sensationson the skin 1500 of a user, e.g., to reproduce effects of someone elsetouching the forearm 1500.

In one embodiment, several frequency sets may be constructed todetermine the robustness and parameter range for pleasant touch atdifferent amplitudes across a broad vibrotactile spectrum. Differentfrequencies trigger different receptive systems within the human body1500 as described and illustrated above with respect to FIG. 15D. Eachfrequency set may include one or more frequencies of vibration atvarying amplitudes. That is, actuator signals including more than onefrequency may be transmitted to the cutaneous actuators to cause thecutaneous actuators to generate sensations that include more than onefrequency. For example, piezoelectric actuators (precision ceramicactuators that convert electrical energy directly into linear motionwith high speed and resolution) may be used to vary the amplitude andfrequency of sensations to create different types of pleasant or calmingtouches and to determine a preferred region 1568 of frequencies andamplitudes.

In one experiment, ten frequency sets were constructed and evaluated.The frequency sets included (1) Frequency Set 1: 20 Hz at 15 dBsensation level (SL); (2) Frequency Set 2: 20 Hz at 30 dB SL; (3)Frequency Set 3: 70 Hz at 15 dB SL; (4) Frequency Set 4: 70 Hz at 30 dBSL; (5) Frequency Set 5: 250 Hz at 15 dB SL; (6) Frequency Set 6: 250 Hzat 30 dB SL; (7) Frequency Set 7: 20 Hz+70 Hz at 20 dB SL; (8) FrequencySet 8: 20 Hz+250 Hz at 20 dB SL; (9) Frequency Set 9: 70 Hz+250 Hz at 20dB SL; and (10) Frequency Set 1: 20 Hz+70 Hz+250 Hz at 25 dB SL.

The amplitudes above are expressed in terms of dB SL, which refers tothe number of dB that the haptic sensations are above the sensationthreshold (i.e., the minimum dB required for perception of the hapticsensations). The 15 dB SL amplitude-at-detection-threshold (aDT) may bedetermined as: aDT×(20 log 15), or aDT×5.6, where the aDT is the lowestthreshold a user can detect. Similarly, the 20 dB SL aDT may bedetermined as: aDT×(20 log 20), or aDT×31.6.

The haptic sensations may be transmitted to several participants of anexperiment, where the first haptic sensations and the second hapticsensations correspond to the one or more frequencies in a frequency set.Each participant may wearing an array of the cutaneous actuators for theexperiment. A rating (e.g., pleasantness) of the continuous tactilemotion may be received from each participant of the experiment for eachfrequency set. In one experiment, each of the ten frequency setsdescribed above were applied to participants at five different speeds ofthe haptic sensations, i.e., 5.7 cm/s, 7.1 cm/s, 9.2 cm/s, 13.2 cm/s,and 23.4 cm/s. The speed refers to the distance along the body 1500 thatthe haptic vibration strokes cover by unit of time. Therefore, for aspeed of 5.7 cm/s, the stroke moves 5.7 cm along the body of theparticipant in one sec. The experiment included 50 patterns, i.e., 5speeds×10 frequency sets. The experiment was performed on 21participants (11 males, average age: 37 years old) who rated the 50patterns on a pleasantness scale of between −7 (less pleasant) and +7(more pleasant).

Similarly, for each amplitude of a set of amplitudes of vibration, thefirst haptic sensations and the second haptic sensations may betransmitted to each of several participants, where the first hapticsensations and the second haptic sensations correspond to the amplitude.A rating (e.g., pleasantness or continuity of the tactile motion) may bereceived from each participant for the amplitude. The received ratingsmay be aggregated to determine a preferred amplitude and frequency setcorresponding to a highest rating.

In one experiment, a preferred frequency of the haptic sensations wasdetermined to equal or be greater than 20 Hz. The preferred frequency ofthe haptic sensations was determined to equal or be less than 70 Hz. Apreferred amplitude of the haptic sensations was determined to equal orbe greater than 15 dB SL. The preferred amplitude of the hapticsensations was determined to equal or be less than 30 dB SL. In otherembodiments, the amplitude of the haptic sensations may lie between 15dB SL and 30 dB SL.

In one experiment, a preferred region of operation 1568 for thecutaneous actuators was determined from the ratings provided by theparticipants. The preferred region of operation 1568 included afrequency of 70 Hz and an amplitude of 15 db SL within a usable range of20-70 Hz and 15-20 dB SL, as depicted in FIG. 15E. Higher amplitudeswere determined to be less pleasant while lower amplitudes were morepleasant.

In one experiment, when the cutaneous actuators generated hapticsensations including more than one frequency, it was determined thatcombining equal components of low frequencies and high frequencies(e.g., 20 Hz+250 Hz) rendered the touch less pleasant. However,combining a larger low-frequency component with a smaller high-frequencycomponent (e.g., 20 Hz+70 Hz+250 Hz) was determined to offset the highfrequency component and render the touch more pleasant.

In one experiment, lower frequencies (e.g., 20 Hz) were rated positivelyon the pleasantness scale, i.e., less than 5% of users reported lesspleasant sensations. Higher amplitudes were rated less pleasant, i.e.,less than 5% of users reported pleasant sensations. Female participantstended to rate the haptic sensations higher on the pleasantness scale.

A preferred region of operation 1568 for the cutaneous actuators 1508may therefore be constructed in terms of frequency range and amplituderange. In one experiment, frequencies of 20-70 Hz at 15-30 dB SL weredetermined to be a preferred range 1568 of the parameters.

Similarly, preferred spacing distances 1512A and 1512B between thecutaneous actuators may be determined. For each of a set of spacingdistances between the first cutaneous actuator 1508A and the secondcutaneous actuator 1508B, the first haptic sensations and the secondhaptic sensations may be transmitted to each of several participants.For each of a set of spacing distances between the cutaneous actuator1508C and the cutaneous actuator 1508D, the haptic sensations may betransmitted to each of several participants. A rating of the continuoustactile motion may be received from each participant for the spacingdistance 1512A or 1512B. The received ratings may be aggregated todetermine a preferred spacing distance 1512A or 1512B corresponding to ahighest rating.

Similarly, a preferred overlap time may be determined between thegenerating of the first haptic sensations and the generating of thesecond haptic sensations. The haptic sensations may be transmitted toeach participant of a set of participants for each of several distinctoverlap times between the generating of the first haptic sensations andthe generating of the second haptic sensations. A rating (e.g.,pleasantness or continuity) of the continuous tactile motion may bereceived from each participant for each overlap time 1536. The receivedratings may be aggregated to determine a preferred overlap timecorresponding to a highest rating.

In one experiment, a preferred overlap time between the generating ofthe first haptic sensations and the generating of the second hapticsensations was determined. Ratings of continuity of the tactile motionversus overlap time 1536 were aggregated. The number 1 was used byparticipants to rate the tactile motion between the cutaneous actuatorsas being more discontinuous, while the number 7 was used to rate themotion as being more continuous. A preferred overlap time was determinedbetween successive vibration locations that generated smooth-continuousillusory strokes on the hairy skin of the forearm using a range offrequencies, amplitudes, and duration times (t3−t1). The duration timeis the time period for which a cutaneous actuator is on. A longerduration time provides a longer touch. Six participants (five males)rated 120 different haptic vibration patterns on a scale of 1-7 forcontinuity. The stimulus used was (1) one frequency set: 20 Hz+70 Hz+250Hz; (2) two different amplitudes (15 dB SL and 30 dB SL); (3) fourdifferent duration times (100 ms, 400 ms, 900 ms, and 1600 ms); and (4)five different overlap times expressed as a percentage of the durationtime that each cutaneous actuator was turned on (20%, 35%, 50%, 65%, and80% of duration time). In other a frequency of the haptic sensations maylie between 1 Hz and 300 Hz.

A psychophysical model including a preferred overlap time and preferredduration time (d) was determined for more continuous and pleasant touch.In one embodiment, the preferred overlap time of the haptic sensationsgenerated by consecutive cutaneous actuators increases as a durationtime of the haptic sensations generated by each cutaneous actuatorincreases. For example, the preferred overlap time of the first hapticsensations and the second haptic sensations may be a function of 0.169times a duration time of the first haptic sensations. The overlap timemay be determined as a function of the duration time (d): 0.169×d+k,where k is a constant. In one example, k may equal 46.05. The preferredoverlap time for more pleasant touches was determined to vary withduration time d, but not as much with frequency or amplitude.

In one embodiment, the arrangement of the cutaneous actuators 1508 andthe algorithms that drive the cutaneous actuators may be varied. In oneembodiment, a haptic illusion at a location E on the user's body 1500may be provided by cutaneous actuators at other locations B and C on theuser's body 1500. Separate spectral components of haptic sensations maythus be combined to create pleasing touch motion. In one embodiment, thefrequency, size, timing, and voltage of a periodic signal used to driveeach cutaneous actuator may be varied. In one experiment, a reactiontime of less than 100 ms (from the time the actuator signals are sent tothe array 1502 to the time the cutaneous actuators turn on) wasdetermined to be pleasant. In one embodiment, the actuator signals mayinclude information on pressure, temperature, texture, sheer stress,time, and space of the physical touch. The amplitude and frequency ofthe haptic sensations described above with respect to FIG. 15E may berelated to the pressure, texture, and sheer stress experienced by thebody 1500 of the user when receiving the haptic sensations.

Cutaneous Actuators with Dampening Layers and End Effectors to IncreasePerceptibility of Haptic Signals

Embodiments also relate to a wearable haptic communication deviceincluding cutaneous actuators and padding layers to dampen hapticvibrations generated by the cutaneous actuators. The hapticcommunication device includes cutaneous actuators to generate hapticvibrations corresponding to actuator signals received by the cutaneousactuators. A dampening member, proximate to a body of a user wearing thehaptic communication device, focuses the haptic vibrations at distinctlocations on the body. The dampening member has openings through whichthe cutaneous actuators transmit the haptic vibrations to the distinctlocations. A spacing member contacts the dampening member and isseparated from the body by the dampening member. The spacing member hasopenings dimensioned to receive and secure the cutaneous actuators.

FIG. 16A is a cross sectional view of an example haptic communicationdevice 1600, in accordance with an embodiment. The haptic communicationdevice 1600 includes an array of cutaneous actuators 1602, a dampeningmember 1604, a spacing member 1606, electronic circuitry 1608, adampening member 1610, a cushioning member 1612, end effectors 1620, anda rigid substrate 1614. In other embodiments, the haptic communicationdevice 1600 may include additional or fewer components than thosedescribed herein. Similarly, the functions can be distributed among thecomponents and/or different entities in a different manner than isdescribed here.

The array of cutaneous actuators 1602 generate haptic vibrations 1616corresponding to actuator signals received by the cutaneous actuators1602. The actuator signals may be voltage signals, current signals, someother transducer signals, or a combination thereof. The actuator signalsmay be generated by various methods as described with respect to FIGS.9F, 14A, 14C, 17A, and 18A. The haptic vibrations 1616 communicatehaptic output (e.g., words in a haptic lexicon) to a body 1618 of a uservia the skin when the user is wearing the haptic communication device1600. The communication of the haptic output may be performed by variousmethods as described with respect to FIGS. 8C, 9A, 10E, 15B, and 15C. Asdescribed above with respect to FIG. 1, the cutaneous actuators may bevoice coils, linear resonant actuators (LRAs), eccentric rotating massactuators, piezoelectric actuators, electroactive polymer actuators,shape memory alloy actuators, pneumatic actuators, microfluidicactuators, or acoustic metasurface actuators.

The dampening member 1604 shown in FIG. 16A is a portion of the hapticcommunication device 1600 that dampens the haptic vibrations 1616. Thedampening member 1604 is proximate to the body 1618 of the user wearingthe haptic communication device 1600. In one embodiment (illustrated anddescribed with respect to FIG. 16D below), the dampening member 1604 maydirectly contact the body 1618 of the user wearing the hapticcommunication device 1600. In another embodiment (illustrated anddescribed with respect to FIG. 16G below), the dampening member 1604 maybe enclosed within a housing, which directly contacts the body 1618 ofthe user. In this embodiment, the dampening member is located between aroof of the housing and the cutaneous actuator, which is also enclosedby the housing.

By dampening the haptic vibrations 1616 generated by the cutaneousactuators 1602, the dampening member 1604 focuses the haptic vibrations1616 at distinct locations on the body. In FIG. 16A, the dampeningmember 1604 focuses the haptic vibrations 1616 at the distinct locationon the body 1618 contacting the end effector 1620. The dampening member1604 contacts skin around the end effector 1620 to dampen the vibrations1616 propagating down the skin for improved localization. “Durometer” isone of several measures of the hardness of a material. The dampeningmember 1604 is stiffer (e.g., durometer 20-50) than the cushioningmember 1612 and serves as an interface between the stiff spacing member1606 and the skin.

In embodiments, the dampening member 1604 may be a layer of soft rubberor polyurethane foam having a thickness of about one sixteenth inch. Thedampening member 1604 provides comfort to the user wearing the hapticcommunication device 1600. The dampening member 1604 is firm, easy toclean, and does not propagate the haptic vibrations 1616. This makes itideal for the layer that touches the body 1618.

Openings 1668 are formed in the dampening member 1604 (shown below inFIG. 16D). The cutaneous actuators 1602 transmit the haptic vibrations1616 to the distinct locations on the body 1618 through the openings1668.

In an embodiment, the haptic communication device 1600 may include asecond dampening member 1610 to dampen vibrations of the cutaneousactuators 1602. The dampening member 1610 contacts the cushioning member1612. The dampening member 1610 is stiffer (e.g., durometer 20-50) thanthe cushioning member 1612 and serves as an interface between the stiffspacing member 1606 and the compliant cushioning member 1612. In anembodiment, the dampening member 1610 may be a layer of soft rubber orpolyurethane foam having a thickness of about one sixteenth inch. In anembodiment, the dampening member 1610 may be made of a quick recoveryfoam.

The spacing member 1606 shown in FIG. 16A supports and holds thecutaneous actuators 1602 in place within the haptic communication device1600. The spacing member 1606 contacts the dampening member 1604 and isseparated from the body 1618 by the dampening member 1604, as shown inFIG. 16A. The spacing member 1606 may be made of stiff rubber to createa solid stack between the cutaneous actuators 1602 and the user's body1618 to ensure that the cutaneous actuator 1602 is able to move againstthe user's skin and is not overly compressed during mounting. The stackis also sometimes referred to as “tolerance stack” and holds thecutaneous actuators 1602 up at a certain distance above the skin so thatthe magnet and coils of the cutaneous actuators 1602 can go throughtheir range of motion and are not being pressed such that they are stuckaway from the magnet center.

In an embodiment, the cutaneous actuators 1602 may be attached to arigid curved surface of the spacing member 1606 and pressed into theskin to ensure that all the cutaneous actuators 1602 make contact withthe skin. The consistent stack spacing ensures that the cutaneousactuators 1602 have reduced load and are able to move through theirrange of motion, more efficiently converting electrical energy tomechanical energy.

The spacing member 1606 has openings (shown below in FIG. 16B) that aredimensioned to receive and secure the cutaneous actuators 1602. Aportion of each cutaneous actuator 1602 is therefore embedded in thespacing member 1606. In embodiments, the spacing member is made of alayer of rigid rubber having a thickness of about one quarter inch and adurometer of less than 50. The durometer value of the spacing member1606 is selected to enable the spacing member 1606 to support thecutaneous actuators 1602 in place within the haptic communication device1600 and make the spacing member 1606 generally resistant toindentation.

The haptic communication device 1600 includes electronic circuitry 1608coupled to the cutaneous actuators 1602 to drive the cutaneous actuators1602. The electronic circuitry 1608 is located within a space 1622between the cushioning member 1612 and the spacing member 1606. Thespace 1622 includes the wire routing and locating layers to locate thecutaneous actuators 1602. The electronic circuitry 1608 may be digitallogic, analog circuitry, or circuitry implemented on a printed circuitboard (PCB) that mechanically supports and electrically connectselectronic components and the cutaneous actuators 1602 using wires,conductive tracks, pads, etc. The haptic communication device 1600includes clearance to cover the electronic circuitry 1608 and improveaesthetics of the haptic communication device 1600. In one embodiment,the cutaneous actuator 1602 is attached to a PCB that has a sensor and aconnector; the space 1622 between the spacing member 1606 and thedampening member 1610 gives the PCB room to be rigid and not bend withthe other layers. The electronic circuitry 1608 transmits the actuatorsignals to the cutaneous actuators 1602. The electronic circuitry 1608may also determine a temperature of the haptic communication device 1600using a temperature sensor (e.g., a thermistor, a thermocouple, aresistance thermometer, or a silicon bandgap temperature sensor). Theelectronic circuitry 1608 may measure the temperature onboard the hapticcommunication device 1600 and transmit corresponding measurement signalsto a main electronic system offboard. Responsive to the temperatureexceeding a threshold, the electronic circuitry 1608 may terminateoperation of the cutaneous actuators. For example, if the temperatureincreases to uncomfortable levels, the main electronic system may cutpower to the cutaneous actuator 1602 via the electronic circuitry 1608.

The cushioning member 1612 shown in FIG. 16A is designed to providephysical comfort to the body 1618 of the user wearing the hapticcommunication device 1600. The compliant cushioning member 1612 adjuststo the arm shape of the user to ensure contact and comfort. Thecushioning member 1612 enables the same haptic communication device 1600to be used on different shaped arms of different users. The cushioningmember 1612 has a low durometer (e.g., durometer 1-10) and deforms toadapt to the user's body 1618.

The cushioning member 1612 adjusts to the shape of the body of the userby compression when the user is wearing the haptic communication device1600. The spacing member 1606 is therefore located between the dampeningmember 1604 and the cushioning member 1612. The cushioning member 1612has openings 1613, and each cutaneous actuator 1602 is inserted into acorresponding one of the openings 1613. A portion of each cutaneousactuator 1602 is embedded in the cushioning member 1612 as shown in FIG.16A. The cushioning member 1612 may be a layer of polyurethane foamhaving a thickness of about one quarter inch to enable compression ofthe cushioning member 1612.

The end effectors 1620 transmit the haptic vibrations 1616 to the body1618 of the user wearing the haptic communication device 1600. The endeffector 1620 touches the body 1618 of the user. Each end effector 1620presses into the body 1618 and focuses the haptic vibrations 1616 at adistinct location on the body 1618 by directing the haptic vibrations1616 along the end effector 1620 from the broader base of the endeffector 1620 mounted on the cutaneous actuator 1602 to the narrower end1638 of the end effector 1620. Each end effector 1620 is mounted on acorresponding one of the cutaneous actuators 1602 to provide comfort tothe body 1618 of the user while generating the haptic vibrations 1616.In an embodiment, smaller end effectors 1620 are easier to localize onthe body 1618. The end effector 1620 concentrates the energy from thecutaneous actuator 1602 into a small area on the user's body 1618. Theend effectors 1620 may be made of rubber, silicone, or polycarbonate. Inan embodiment, the end effector 1620 may have a deburred acrylic end1638.

The haptic communication device 1600 includes a rigid substrate 1614that is dimensioned to compress the cushioning member 1612 to the shapeof the body 1618. The cushioning member 1612 is therefore locatedbetween the spacing member 1606 and the rigid substrate 1614. In oneembodiment, the rigid substrate 1614 may be dimensioned similar to ashin guard and made of similar material. In one embodiment, theassembled dampening and cushioning members may be attached (e.g.,double-sided taped) to an Acrylonitrile Butadiene Styrene (ABS) plasticcurved sheet that serves as the rigid substrate 1614, to which strapsand other electronics are also attached.

In some embodiments, the substrate 1614 may be made of molded plastic,machined plastic, or 3D-printed plastic. In some embodiments, thesubstrate 1614 may be made of VeroWhite, nylon, ABS, or polycarbonate.In some embodiments, the substrate 1614 may be machined out ofpolycarbonate or nylon, or 3-D printed ABS.

FIG. 16B is a perspective view of components of the example hapticcommunication device 1600, in accordance with an embodiment. The hapticcommunication device 1600 shown in FIG. 16B includes the spacing member1606 and the cushioning member 1612. The haptic communication device1600 shown in FIG. 16B is shaped and dimensioned to support fourcutaneous actuators 1602, as shown by the four openings 1624. In otherembodiments, the haptic communication device 1600 may include additionalor fewer components than those described herein. Similarly, thefunctions can be distributed among the components and/or differententities in a different manner than is described here.

The spacing member 1606 has openings 1624 that are dimensioned toreceive and secure the cutaneous actuators 1602 and the electroniccircuitry 1608. Each of the openings 1624 has a first portion 1626having a first shape, and a second portion 1628 extending from the firstportion and having a second shape. The cutaneous actuators 1602 areinserted into the one or more first portions 1626, and the electroniccircuitry 1608 is inserted into the one or more second portions 1628.For example, each of the openings 1624 may have a key-hole shape havinga circular portion 1626 and a rectangular portion 1628 extending fromthe circular portion 1626. The rectangular portion 1628 providesclearance for the electronic circuitry 1608 and wiring. The cutaneousactuators 1602 are inserted into the circular portions 1626 and theelectronic circuitry 1608 is inserted into the rectangular portions 1628of the openings 1624.

The cushioning member 1612 made of polyurethane foam having a thicknessof about one quarter inch provides physical comfort to the body 1618 ofthe user wearing the haptic communication device 1600. The electroniccircuitry 1608 is located within the space 1622 between the cushioningmember 1612 and the spacing member 1606.

FIG. 16C is a perspective view of the example haptic communicationdevice 1600 mounted with cutaneous actuators 1602, in accordance with anembodiment. The haptic communication device 1600 shown in FIG. 16Cincludes the spacing member 1606 and the cushioning member 1612. Inother embodiments, the haptic communication device 1600 may includeadditional or fewer components than those described herein. Similarly,the functions can be distributed among the components and/or differententities in a different manner than is described here.

The haptic communication device 1600 shown in FIG. 16C is shaped anddimensioned to support four cutaneous actuators 1602. The cutaneousactuators 1602 are shown inserted into the circular portions 1626 of theopenings 1624. In FIG. 16C, the electronic circuitry 1608 is showninserted into the rectangular portions 1628 of the openings 1624. Theelectronic circuitry 1608 is located within the space 1622 between thecushioning member 1612 and the spacing member 1606.

FIG. 16D is a perspective view of a portion of the example hapticcommunication device 1600 including a dampening member 1604, inaccordance with an embodiment. The haptic communication device 1600shown in FIG. 16D includes the dampening member 1604, the spacing member1606, the second dampening member 1610, the cushioning member 1612, andthe end effectors 1620. The haptic communication device 1600 shown inFIG. 16D is shaped and dimensioned to support four cutaneous actuators1602, as shown by the four end effectors 1620. In other embodiments, thehaptic communication device 1600 may include additional or fewercomponents than those described herein. Similarly, the functions can bedistributed among the components and/or different entities in adifferent manner than is described here.

The dampening member 1604 has four openings 1668 as shown in FIG. 16D.The cutaneous actuators 1602 transmit the haptic vibrations 1616 to thedistinct locations on the body 1618 through the openings 1668. The sizeof the openings may be designed to surround the end effector 1620. Inone embodiment, the openings may be around 1 inch in diameter. Thebenefit of the openings is that they restrict the haptic vibrations 1616to stay within the localized area, and result in better localization andperceptibility. The electronic circuitry 1608 is located within thespace 1622 between the cushioning member 1612 and the spacing member1606.

Embodiments also relate to a haptic communication device 1630(illustrated and described in detail below with respect to FIG. 16E)including housings mounted on a rigid substrate worn by a user andcutaneous actuators 1602 that are enclosed and separated from each otherby each housing. The haptic communication device 1630 includes acutaneous actuator 1602, at least a part of which is located within ahousing to generate haptic vibrations 1616 corresponding to actuatorsignals received by the cutaneous actuator 1602. A dampening member islocated within the housing and between the cutaneous actuator 1602 and aroof of the housing. The dampening member focuses the haptic vibrations1616 at a distinct location on a body 1618 of a user wearing the hapticcommunication device 1630. The dampening member has a first opening,wherein the cutaneous actuator 1602 transmits the haptic vibrations tothe distinct location through the first opening. A spacing member islocated within the housing and contacts the dampening member. Thespacing member is separated from the body 1618 by the dampening memberand has a second opening dimensioned to receive and secure the cutaneousactuator 1602.

FIG. 16E is a perspective view of an example haptic communication device1630 including a cutaneous actuator 1602 partially located within ahousing 1632, in accordance with an embodiment. The haptic communicationdevice 1630 includes an array of cutaneous actuators 1602. Eachcutaneous actuator 1602 is inserted into a housing 1632 on the array.The portion of the haptic communication device 1630 shown in FIG. 16Eincludes the housing 1632, the cutaneous actuator 1602, the electroniccircuitry 1608, and the end effector 1620. In other embodiments, thehaptic communication device 1600 may include additional or fewercomponents than those described herein. Similarly, the functions can bedistributed among the components and/or different entities in adifferent manner than is described here.

The housing 1632 is a rigid casing that encloses and protects a portionof the cutaneous actuator 1602 and the electronic circuitry 1608 coupledto the cutaneous actuator 1602. The housing 1632 directly contacts thebody 1618 of a user wearing the haptic communication device 1630. Thehousing 1632 may be in the shape of a cylinder, a cone, a cube, a prism,etc. The cutaneous actuator 1602 is partly located within the housing1632 to generate haptic vibrations 1616 corresponding to actuatorsignals received by the cutaneous actuator 1602. The actuator signalsmay be generated by various methods as described with respect to FIGS.9F, 14A, 14C, 17A, and 18A. The haptic vibrations 1616 communicatehaptic output (e.g., words in a haptic lexicon) to a body 1618 of a uservia the skin when the user is wearing the haptic communication device1600. The communication of the haptic output may be performed by variousmethods as described with respect to FIGS. 8C, 9A, 10E, 15B, and 15C. Asdescribed above with respect to FIG. 1, the cutaneous actuator 1602 maybe a voice coil, a linear resonant actuator (LRA), an eccentric rotatingmass actuator, a piezoelectric actuator, an electroactive polymeractuator, a shape memory alloy actuator, a pneumatic actuator, amicrofluidic actuator, or an acoustic metasurface actuator.

The housing 1632 for each cutaneous actuator 1602 in the array ofcutaneous actuators may be mounted on a rigid substrate 1614 (shownbelow in FIG. 16H) to form the haptic communication device 1630. Thehousing 1632 increases localization of haptic effects on the body 1618by focusing the haptic vibrations 1616 on the body 1618. In someembodiments, the housing 1632 may be made of molded plastic, machinedplastic, or 3D-printed plastic. For example, the housing 1632 may bemade by using a mold that is filled with pliable plastic material. Theplastic hardens or sets inside the mold, adopting its shape. In anotherexample, the housing 1632 may be made by taking a piece of plastic andcutting into a desired final shape and size by a controlledmaterial-removal process (machining). In another example, the housing1632 may be made by joining or solidifying plastic material undercomputer control to create the three-dimensional housing 1632 (3Dprinting).

In some embodiments, the housing 1632 may be made of VeroWhite, nylon,Acrylonitrile Butadiene Styrene (ABS), or polycarbonate. In someembodiments, the housing 1632 may be machined out of polycarbonate ornylon, or 3-D printed ABS for ease of manufacturing and biocompatibilitywith human skin. VeroWhite is an opaque Polyjet resin that is used whenprototyping new designs and can also be used for creating end-useproducts. ABS is a two phase polymer blend having rigidity, hardness,and heat resistance. Polycarbonate is a synthetic resin in which thepolymer units are linked through carbonate groups, and is used formolding materials and films.

A roof 1634 of the housing 1632 provides support to the hapticcommunication device 1630. A portion of the roof 1634 may rest against abody 1618 of the user when the haptic communication device 1630 istransmitting the haptic vibrations 1616 to the user. The roof 1634 hasan opening 1636 for an end effector 1620 to contact the body of the user1618.

The end effector 1620 presses into the body 1618 of the user andtransmits the haptic vibrations 1616 to the user. The end effector 1620extends out of the housing 1632 through the opening 1636 in the roof1634. The end effector 1620 is mounted on the cutaneous actuator 1602and focuses the haptic vibrations 1616 at a distinct location on thebody 1618 while providing a comfortable experience to the user. Forexample, the end effector 1620 protects the user's body from coming intocontact with parts of the cutaneous actuator 1602 or housing 1632 thatmay be rigid and hurt the user.

The end effector 1620 has one end (shown as end 1642 below in FIG. 16G)attached to the cutaneous actuator 1602. Another end 1638 of the endeffector 1620 contacts the body 1618 of the user. The end 1642 has asurface area larger than a surface are of the end 1638, such that theend effector 1620 may increase pressure of the haptic vibrations 1616 onthe body 1618. In embodiments, the end effector 1620 may be made ofrubber, silicone, or polycarbonate. The end effector 1620 may have adisc-like shape, a cylindrical shape, a tapered shape, or a dome-likeshape.

The electronic circuitry 1608 is inserted into the housing 1632 alongwith the cutaneous actuator 1602 to power the cutaneous actuator 1602and transmit the actuator signals to the cutaneous actuator 1602 tocause the cutaneous actuator 1602 to generate the haptic vibrations1616. The electronic circuitry 1608 is located within a space betweenthe cutaneous actuator 1602 and an inner wall of the housing 1632 (e.g.,inner wall 1658 shown below in FIG. 16G). In one embodiment, theelectronic circuitry 1608 is a PCB and includes a temperaturemeasurement sensor. The electronic circuitry 1608 may measure theonboard temperature of the haptic communication device 1630 and transmitcorresponding measurement signals to a main electronic system offboard.Responsive to the temperature exceeding a threshold, the electroniccircuitry 1608 may terminate operation of the cutaneous actuators. Forexample, if the temperature increases to uncomfortable levels, a mainelectronic system may cut power to the cutaneous actuator 1602 via theelectronic circuitry 1608.

FIG. 16F is a perspective view of the example haptic communicationdevice 1630 including a centering member 1640, in accordance with anembodiment. FIG. 16F includes the housing 1632, a portion of the roof1634, electronic circuitry 1608, and the centering member 1640. In otherembodiments, the haptic communication device 1600 may include additionalor fewer components than those described herein. Similarly, thefunctions can be distributed among the components and/or differententities in a different manner than is described here.

The haptic communication device 1630 shown in FIG. 16F includes thecentering member 1640 to protect the cutaneous actuator 1602 and the endeffector 1620 from rubbing against the inner walls of the housing 1632.The centering member 1640 locates the cutaneous actuator 1602 within thehousing 1632. The centering member 1640 is located within the housing1632 at the bottom of the housing 1632, as shown in FIG. 16G below. Thecentering member 1640 surrounds a portion of the cutaneous actuator 1602and constrains the cutaneous actuator 1602 and the end effector 1620 ina plane parallel to the roof 1634.

The centering member 1640 constrains the cutaneous actuator 1602 and theend effector 1620 in the X-Y plane to ensure that the cutaneous actuator1602 and the end effector 1620 are not off-center or rubbing against thewalls of the housing 1632. The centering member 1640 thus prevents thecutaneous actuator 1602 and the end effector 1620 from contacting aninner wall of the housing 1632. Thus, the centering member 1640 preventswastage of energy from the cutaneous actuator 1602 or the end effector1620 in shaking the housing 1632 or causing friction by rubbing againstthe housing 1632. If the cutaneous actuator 1602 were biased off-center,there would be a significant non-Z axis force that would be wasted; thisis prevented by the disclosed embodiments herein. The centering member1640 may be made of materials similar to the materials of the housing1632.

FIG. 16G is a cross sectional view of the example haptic communicationdevice 1630 taken along the X axis of FIG. 16F, in accordance with anembodiment. FIG. 16G includes the housing 1632, a portion of the roof1634, electronic circuitry 1608, the centering member 1640, the endeffector 1620, and the cutaneous actuator 1602.

The housing 1632 is the rigid casing that encloses and protects thecutaneous actuator 1602 and the electronic circuitry 1608. An end 1642of the end effector 1620 is mounted on the cutaneous actuator 1602 totransmit the haptic vibrations 1616 generated by the cutaneous actuator1602 to the body 1618 of the user wearing the haptic communicationdevice 1630. Another end 1638 of the end effector 1620 contacts orpresses into the user's body 1618 to transmit the haptic vibrations1616. The end 1642 has a surface area larger than a surface area of theend 1638, such that the end effector 1620 may increase pressure of thehaptic vibrations 1616 on the body 1618. The end effector 1620 mayextend outwards out of the housing 1632 through the opening 1636 tocontact the user's body 1618.

The centering member 1640 surrounds and protects the cutaneous actuator1602 and the end effector 1620 from rubbing against the inner walls 1646of the housing 1632, as described above with respect to FIG. 16F. Theelectronic circuitry 1608 may be inserted along with the cutaneousactuator 1602 into the housing 1632 through an opening in the walls ofthe housing 1632 as shown in FIG. 16G.

The area 1644 within the housing 1632 to the side of the cutaneousactuator 1602 in FIG. 16G may be used to contain a dampening member 1604(as described and shown above with respect to FIG. 16A) that is locatedwithin the housing 1632. The dampening member 1604 is located betweenthe cutaneous actuator 1602 and the roof 1634 of the housing 1632. Thedampening member 1604 dampens the haptic vibrations 1616 generated bythe cutaneous actuator 1602 and focuses the haptic vibrations 1616 at adistinct location on the body 1618 of the user wearing the hapticcommunication device 1630.

The dampening member 1604 dampens the haptic vibrations 1616 generatedby the cutaneous actuator 1602 to focus the haptic vibrations 1616 at adistinct location on the body 1618. In embodiments, the dampening member1604 may be a layer of soft rubber or polyurethane foam having apredetermined thickness (e.g., about one sixteenth inch). The dampeningmember has an opening (as shown above in FIG. 16D); the cutaneousactuator 1602 transmits the haptic vibrations 1616 to the distinctlocation on the body 1618 through the opening.

The area 1644 in FIG. 16G may also be used to contain a spacing member1606 (as illustrated and described above with respect to FIG. 16A) thatis located within the housing 1632. The spacing member 1606 contacts thedampening member 1604. The spacing member 1606 is separated from thebody 1618 of the user by the dampening member 1604 and has an opening1624 (as illustrated and described above with respect to FIG. 16B) thatis dimensioned to receive and secure the cutaneous actuator 1624. Aportion of the spacing member 1606 is located between the dampeningmember 1604 and a section 1646 of the cutaneous actuator 1602 extendingoutward from a center of the cutaneous actuator 1602 to position thecutaneous actuator 1602 such that the cutaneous actuator 1602 does notrub against or contact the roof 1634.

In embodiments, the spacing member 1606 is made of a layer of rigidrubber having a predetermined thickness (e.g., about one quarter inch)and a durometer of less than 50. The durometer value of the spacingmember 1606 is selected to enable the spacing member 1606 to support thecutaneous actuators 1602 in place within the housing 1632 and make thespacing member 1606 generally resistant to indentation.

The haptic communication device 1630 has similar Z-stack spacing andvibration dampening features to the haptic communication device 1600shown above in FIG. 16A. The haptic communication device 1630 also hasconical features to self-center the end effector 1620 during mounting toprevent the end effector from rubbing on the housing 1632.

FIG. 16H is a perspective view of the example haptic communicationdevice 1630 including the rigid substrate 1614, in accordance with anembodiment. The haptic communication device 1630 (as shown in FIG. 16H)includes an array of six cutaneous actuators 1602. Each cutaneousactuator is enclosed within the housing 1632, such that it may generateand transmit the haptic vibrations 1616 to the body 1618 of the user. Aportion of the electronic circuitry 1608 is also inserted within thehousing 1632. The haptic communication device 1630 includes the rigidsubstrate 1614 on which each housing 1632 is mounted. The rigidsubstrate 1614 (also described above with respect to FIG. 16A) isdimensioned to secure the housing 1632 to a shape of the body 1618 ofthe user. In one embodiment, each housing 1632 may be double-sided tapedto an ABS plastic curved sheet that serves as the rigid substrate 1614,to which straps and other electronics are also attached.

In one embodiment, a haptic communication system may be constructed,including a signal generator configured to receive a message from asending user and generate actuator signals corresponding to the message.The system includes the haptic communication device 1630 communicativecoupled to the signal generator. The haptic communication device 1630includes a cutaneous actuators 160 configured to generate the hapticvibrations 1616 corresponding to the actuator signals to transmit themessage to the body 1618 of a user wearing the haptic communicationdevice 1630. The haptic communication device 1630 includes the housing1632 in direct contact with the body 1618 of the user and at leastpartially enclosing the cutaneous actuator 1602. The dampening member1604 is located within the housing 1632 and proximate to the body 1618to focus the haptic vibrations 1616 at a distinct location on the body1618. The dampening member 1604 has an opening, wherein the cutaneousactuator 1602 transmits the haptic vibrations 1616 to the distinctlocation through the opening. A spacing member 1606, also located withinthe housing 1632, contacts the dampening member 1604 and is separatedfrom the body 1618 by the dampening member 1604. The spacing member 1606has an opening dimensioned to receive and secure the cutaneous actuators1602.

FIGS. 16I-16L are perspective views of different end effectors, inaccordance with an embodiment. Each type of end effector may be mountedon the top of the cutaneous actuator 1602 as shown in FIG. 16A. Thedifferent types of end effectors have different dimensions and may bemade of different materials to achieve different contact areas and/orstiffness. The end effectors have at least three benefits. First, theyprovide consistent contact between the cutaneous actuator 1602 and thebody 1618 during use of the haptic communication device 1600 when thecutaneous actuator 1602 vibrates and when the user's body 1618 moves.Second, they may be made of soft rubber, silicone or polycarbonatepadding material so they do not irritate and/or cut/hurt users. Third,they translate and focus the actuator energy to the skin.

FIG. 16I is a perspective view of a cylindrical end effector, inaccordance with an embodiment. The cylindrical end effector shown inFIG. 16I has a cylindrical tip 1648 positioned on top of the disc 1650.The end 1638 on top of the cylindrical tip 1648 contacts the body 1618of the user to transmit the haptic vibrations 1616 from the cutaneousactuator 1602 to the body 1618. The disc 1650 is positioned on top ofthe base 1652. The tip 1648, disc 1650, and base 1652 may be a singlepiece or three different pieces attached together to make the endeffector. The end 1642 at the bottom of the base 1652 is mounted on thecutaneous actuator 1602 as shown in FIG. 16G. The end 1638 is smaller insurface area than the end 1642 to increase pressure on the body 1618 ofthe user. A benefit of the cylindrical end effector is that it extendsthe space between actuator 1602 and skin for packaging and mounting.

FIG. 16J is a perspective view of a tapered end effector, in accordancewith an embodiment. The tapered end effector shown in FIG. 16J has atapered tip 1654 positioned on top of the disc 1650. The end 1638 on topof the tapered tip 1654 contacts the body 1618 of the user to transmitthe haptic vibrations 1616 from the cutaneous actuator 1602 to the body1618. The disc 1650 is positioned on top of the base 1652. The tip 1648,disc 1650, and base 1652 may be a single piece or three different piecesattached together to make the end effector. The end 1642 at the bottomof the base 1652 is mounted on the cutaneous actuator 1602 as shown inFIG. 16G. The end 1638 is smaller in surface area than the end 1642 toincrease pressure on the body 1618 of the user. A benefit of the taperedend effector is that it reduces the skin contact area and increases thepressure.

FIG. 16K is a perspective view of a disc end effector, in accordancewith an embodiment. The disc end effector shown in FIG. 16K has a largerupper surface 1666 on top of the disc 1650. The end 1666 contacts thebody 1618 of the user to transmit the haptic vibrations 1616 from thecutaneous actuator 1602 to the body 1618. The disc 1650 is positioned ontop of the base 1652. The disc 1650 and base 1652 may be a single pieceor two different pieces attached together to make the end effector. Theend 1642 at the bottom of the base 1652 is mounted on the cutaneousactuator 1602 as shown in FIG. 16G. A benefit of the disc end effectoris that it locates the haptic vibrations 1616 in a controlled area.

FIG. 16L is a perspective view of a dome end effector, in accordancewith an embodiment. The dome end effector shown in FIG. 16L has a dometip 1656 positioned on top of the disc 1650. The dome tip 1656 contactsthe body 1618 of the user to transmit the haptic vibrations 1616 fromthe cutaneous actuator 1602 to the body 1618. The disc 1650 ispositioned on top of the base 1652. The tip 1656, disc 1650, and base1652 may be a single piece or three different pieces attached togetherto make the end effector. The end 1642 at the bottom of the base 1652 ismounted on the cutaneous actuator 1602 as shown in FIG. 16G. The dometip 1656 is smaller in surface area than the end 1642 to increasepressure on the body 1618 of the user. A benefit of the dome endeffector 1664 is that it provides smooth penetration and an optimalcontact point.

FIG. 16M is a perspective view of an example haptic communication device1670 including a housing 1672 that completely encloses a cutaneousactuator 1602, in accordance with an embodiment. The cutaneous actuator1602 (not shown in FIG. 16M) is completely located within the housing1672. The haptic communication device 1670 may include an array ofcutaneous actuators 1602. Each cutaneous actuator 1602 is completelyinserted into a corresponding housing 1672 on the array. The portion ofthe haptic communication device 1670 shown in FIG. 16M includes thehousing 1672 and an end effector 1620. In other embodiments, the hapticcommunication device 1670 may include additional or fewer componentsthan those described herein. Similarly, the functions can be distributedamong the components and/or different entities in a different mannerthan is described here.

The housing 1672 is a rigid casing that encloses and protects thecutaneous actuator 1602 and electronic circuitry 1608 (not shown in FIG.16M) that is connected to and drives the cutaneous actuator 1602. Thehousing 1672 may be in the shape of a cylinder, a cone, a cube, a prism,etc. The cutaneous actuator 1602 is completely located within thehousing 1672 to generate haptic vibrations 1616 corresponding toactuator signals received by the cutaneous actuator 1602. The actuatorsignals may be generated by various methods as described with respect toFIGS. 9F, 14A, 14C, 17A, and 18A. The haptic vibrations 1616 communicatehaptic output (e.g., words in a haptic lexicon) to a body 1618 of a uservia the skin when the user is wearing the haptic communication device1670. The communication of the haptic output may be performed by variousmethods as described with respect to FIGS. 8C, 9A, 10E, 15B, and 15C. Asdescribed above with respect to FIG. 1, the cutaneous actuator 1602 maybe a voice coil, a linear resonant actuator (LRA), an eccentric rotatingmass actuator, a piezoelectric actuator, an electroactive polymeractuator, a shape memory alloy actuator, a pneumatic actuator, amicrofluidic actuator, or an acoustic metasurface actuator.

The housing 1672 for each cutaneous actuator 1602 in the array ofcutaneous actuators may be mounted on a rigid substrate 1614 (shownabove in FIG. 16H) to form the haptic communication device 1670. Thehousing 1672 increases localization of haptic effects on the body 1618by focusing the haptic vibrations 1616 on the body 1618. In someembodiments, the housing 1672 may be made of molded plastic, machinedplastic, or 3D-printed plastic. In some embodiments, the housing 1672may be made of VeroWhite, nylon, Acrylonitrile Butadiene Styrene (ABS),or polycarbonate. In some embodiments, the housing 1672 may be machinedout of polycarbonate or nylon, or 3-D printed ABS for ease ofmanufacturing and biocompatibility with human skin.

A roof 1674 of the housing 1632 provides support to the hapticcommunication device 1630. A portion of the roof 1674 may rest against abody 1618 of the user when the haptic communication device 1670 istransmitting the haptic vibrations 1616 to the user. The roof 1674 hasan opening 1636 for an end effector 1620 (mounted on the cutaneousactuator 1602) to contact the body of the user 1618.

The end effector 1620 presses into the body 1618 of the user andtransmits the haptic vibrations 1616 to the user. The end effector 1620extends out of the housing 1632 through the opening 1636 in the roof1674. The end effector 1620 is mounted on the cutaneous actuator 1602and focuses the haptic vibrations 1616 at a distinct location on thebody 1618 while providing a comfortable experience to the user. Forexample, the end effector 1620 protects the user's body from coming intocontact with parts of the cutaneous actuator 1602 or housing 1672 thatmay be rigid and hurt the user.

The end effector 1620 has one end (shown as end 1642 below in FIG. 16N)attached to the cutaneous actuator 1602. Another end 1638 of the endeffector 1620 contacts the body 1618 of the user. The end 1642 has asurface area larger than a surface are of the end 1638, such that theend effector 1620 may increase pressure of the haptic vibrations 1616 onthe body 1618.

The electronic circuitry 1608 (not shown in FIG. 16M) is inserted intothe housing 1672 along with the cutaneous actuator 1602 to power thecutaneous actuator 1602. The electronic circuitry 1608 is located withina space between the cutaneous actuator 1602 and an inner wall of thehousing 1672 (e.g., inner wall 1676 shown below in FIG. 16N).

FIG. 16N is a cross sectional view of the example haptic communicationdevice 1670 of FIG. 16M, in accordance with an embodiment. FIG. 16Nincludes the housing 1672, a portion of the roof 1674, electroniccircuitry 1608, a centering member 1640, the end effector 1620, and thecutaneous actuator 1602.

The housing 1672 is the rigid casing that completely encloses andprotects the cutaneous actuator 1602 and the electronic circuitry 1608.An end 1642 of the end effector 1620 is mounted on the cutaneousactuator 1602 to transmit the haptic vibrations 1616 generated by thecutaneous actuator 1602 to the body 1618 of the user wearing the hapticcommunication device 1670. Another end 1638 of the end effector 1620contacts or presses into the user's body 1618 to transmit the hapticvibrations 1616. The end 1642 has a surface area larger than a surfacearea of the end 1638, such that the end effector 1620 may increasepressure of the haptic vibrations 1616 on the body 1618. The endeffector 1620 may extend outwards out of the housing 1672 through theopening 1636 to contact the user's body 1618. The centering member 1640surrounds and protects the cutaneous actuator 1602 and the end effector1620 from rubbing against the inner wall 1676 of the housing 1672, asdescribed above with respect to FIG. 16F.

The area 1644 within the housing 1672 to the side of the cutaneousactuator 1602 in FIG. 16N may be used to contain a dampening member 1604(as described and shown above with respect to FIG. 16A) that is locatedwithin the housing 1672. The dampening member 1604 is located betweenthe cutaneous actuator 1602 and the roof 1674 of the housing 1672. Thedampening member 1604 dampens the haptic vibrations 1616 generated bythe cutaneous actuator 1602 and focuses the haptic vibrations 1616 at adistinct location on the body 1618 of the user wearing the hapticcommunication device 1670.

The area 1644 in FIG. 16N may also be used to contain a spacing member1606 (as illustrated and described above with respect to FIG. 16A) thatis located within the housing 1672. A portion of the spacing member 1606is located between the dampening member 1604 and a section 1646 of thecutaneous actuator 1602 extending outward from a center of the cutaneousactuator 1602 to position the cutaneous actuator 1602 such that thecutaneous actuator 1602 does not rub against or contact the roof 1674.

Envelope Encoding of Speech Signals for Transmission to CutaneousActuators

Embodiments also relate to a haptic communication system to conveyspeech messages to a receiving user. A sending user may utter speechsounds or type in a text message into a speech signal generator thatgenerates speech signals corresponding to the speech sounds or thetextual message. A temporal envelope of the speech signals is used toconvey the speech messages to the receiving user. The temporal enveloperepresents changes in amplitude of the speech signals. Carrier signalshaving a periodic waveform are generated and altered using the changesin amplitude of the speech signals to create actuator signals to conveya representation of the temporal envelope to cutaneous actuators. Thecutaneous actuators generate haptic vibrations based on the actuatorsignals. The haptic vibrations represent the speech sounds or textualmessage that the receiving user is able to understand.

FIG. 17A is an illustration of an example haptic communication system1700 for envelope encoding of speech signals 1706 and transmission toone or more cutaneous actuators 1722, in accordance with an embodiment.The haptic communication system 1700 includes a microphone 1704, akeyboard 1708, a speech synthesizer 1712, an envelope encoder 1716, anda haptic communication device 1720. In other embodiments, the hapticcommunication system 1700 may include additional or fewer componentsthan those described herein. Similarly, the functions can be distributedamong the components and/or different entities in a different mannerthan is described here.

As shown in FIG. 17A, the haptic communication system 1700 includes aspeech signal generator (e.g., microphone 1704 or speech synthesizer1712) to receive speech sounds 1702 uttered by a user or a textualmessage 1710 typed out by the user and generate speech signals 1706 or1714 corresponding to the speech sounds 1702 or the textual message1710. For example, the microphone 1704 may generate the speech signals1706 by digitizing the speech sounds 1702. The microphone 1704 is atransducer that converts the speech sounds 1702 into an electricalsignal. The electrical signal may be an analog signal, a digital signal,or a combination thereof. The microphone 1704 may use any of severaldifferent methods (e.g., a coil of wire suspended in a magnetic field, avibrating diaphragm, or a crystal of piezoelectric material, etc.) toconvert the air pressure variations of the speech sounds 1702 to theelectrical signal. In addition, the microphone may include ananalog-to-digital converter to digitize the electrical signal to thespeech signals 1706. The microphone 1704 transmits the speech signals1706 to the envelope encoder over a wired or a wireless connection.

The keyboard 1708 may be used by the user to type out the textualmessage 1710. The keyboard may be part of a personal computer, asmartphone, a laptop, a tablet, etc. The keyboard transmits the textualmessage 1710 to the speech synthesizer 1712 over a wired or a wirelessconnection.

The speech synthesizer 1712 artificially produces human speech. Forexample, a text-to-speech speech synthesizer may convert normal languagetext into speech; other embodiments of the speech synthesizer 1712 mayrender symbolic linguistic representations such as phonetictranscriptions into speech. As shown in FIG. 17A, the speech synthesizer1712 generates the speech signals 1714 from the textual message 1710.The speech synthesizer 1712 may be implemented in software or hardware.For example, the speech synthesizer 1712 may be part of a personalcomputer (PC), a tablet PC, a set-top box (STB), a smartphone, aninternet of things (IoT) appliance, or any machine capable of executinginstructions that specify actions to be taken by that machine. In oneembodiment, the speech synthesizer 1712 maps the textual message 1710 tostored sound signals. For example, the speech synthesizer 1712 maygenerate the speech signals 1714 by concatenating pieces of recordedspeech signals (sound signals) that are stored in a database. Inalternative embodiments, the speech synthesizer 1712 may incorporate amodel of the vocal tract and other human voice characteristics to createa completely “synthetic” voice output.

The envelope encoder 1716 (illustrated and described in more detailbelow with respect to FIG. 17D) encodes characteristics of the speechsignals (e.g., 1706) to generate the actuator signals 1718. In oneembodiment, the envelope encoder 1716 extracts a temporal envelope fromthe speech signals 1706. The temporal envelope represents changes inamplitude of the speech signals 1706 over time. The envelope encoder1716 generates carrier signals having a periodic waveform. The carriersignals may be sinusoidal or square wave pulse signals. The envelopeencoder 1716 generates the actuator signals 1718 by encoding the changesin the amplitude of the speech signals 1706 from the temporal envelopeinto the carrier signals. The actuator signals 1718 thus carryinformation from the temporal envelope encoded within the carriersignals. The envelope encoder 1716 may be implemented in software orhardware. For example, the envelope encoder 1716 may be part of a PC, atablet PC, an STB, a smartphone, an internet of things (IoT) appliance,or any machine capable of executing instructions that specify actions tobe taken by that machine.

As shown in FIG. 17A, the haptic communication device 1720 receives theactuator signals 1718 and generates and transmits haptic vibrations 1724to transmit information from the speech sounds 1702 or the textualmessage 1720 to the body 1726 of the user wearing the hapticcommunication device 1720. The haptic vibrations 1724 represent thespeech sounds 1702 or the textual message 1720. The haptic communicationdevice 1720 may include an array of cutaneous actuators 1722 thatgenerates the haptic vibrations 1724. The cutaneous actuators 1722 areoperably coupled over wired or wireless connections to the envelopeencoder 1716 to generate the haptic vibrations 1724 based on theactuator signals 1718. The haptic communication device 1720 may includeone or more processing units (e.g., a central processing unit (CPU), agraphics processing unit (GPU), a digital signal processor (DSP), acontroller, a state machine, one or more application specific integratedcircuits (ASICs), one or more radio-frequency integrated circuits(RFICs), or any combination of these) and a memory.

In one embodiment, the cutaneous actuators 1722 may be voice coils thatconvert an analog AC-coupled signal (the actuator signals 1718) toproportional mechanical motion. The cutaneous actuators 1722 may bearranged on a semi-rigid backing that secures them in position againstthe body 1726 with enough pressure that the haptic vibrations 1724 canbe perceived by the user. In other embodiments, the haptic communicationdevice 1720 may include piezoelectric, electroactive polymer, eccentricrotating mass, and linear resonant actuators. Other embodiments of thehaptic communication device 1720 are illustrated and described in detailwith respect to FIGS. 1, 8B, 8E, 9A-9E, 11A, 11I, 12A-12B, 15A and16A-16L among others.

FIG. 17B is an illustration of an example temporal envelope 1728 forencoding of the speech signals 1706 and transmission to the cutaneousactuators 1722, in accordance with an embodiment. The signals shown inFIG. 17B include the speech signals 1706, the temporal envelope 1728,and carrier signals 1734.

As shown in FIG. 17B, the speech signals 1706 are represented as acontinuously varying acoustic waveform that can be transmitted,recorded, and manipulated. In one embodiment, the speech signals 1706may comprise acoustic samples, e.g., a discrete signal x[n], i.e., asequence of real or integer numbers corresponding to the signal samplessampled uniformly at a sampling rate. In one embodiment, the samplingrate for the speech signals 1706 may be 16 kHz, while in otherembodiments, the sampling rate (e.g., for “telephone speech” coding) maybe 8 kHz. In one embodiment, the speech signals 1706 may be representedas a number of amplitude-altered signals representing the outputs of anarray of narrow frequency bands (“Fi”). The speech signals 1706 includehigh-frequency components that are altered by a lower frequency, slowlyevolving temporal envelope 1728 as shown in FIG. 17B.

The temporal envelope 1728 shown in FIG. 17B represents changes in theamplitude of the speech signals 1706. The temporal envelope 1728 is theslow variation in the amplitude (e.g., 1732) of the speech signals 1706over time (e.g., 1730). Temporal information at the output of eachfrequency band Fi of the speech signals 1706 may be separated intotemporal fine structure (the rapid oscillations close to the centerfrequency) and the temporal envelope 1728 (the slower amplitude changes)shown in FIG. 17B. The temporal envelope 1728 evolves more slowly intime than the temporal fine structure and requires a lower sampling rateto be represented. The temporal envelope 1728 is a one-dimensionalsignal and can be represented with a single time-varying waveform.

The temporal envelope 1728 may be sufficient for speech comprehension,therefore including sufficient information for transcutaneous speechtransmission using the cutaneous actuators 1722. For example,comprehension suffers when the temporal envelope 1728 is degraded butnot when the fine structure is. Words in speech may therefore beidentified according to the temporal envelope 1728. When the speechsignals 1706 are decomposed into frequency bands and the temporalpatterns in each band are replaced by noise that is altered by thetemporal envelope of each band, the speech remains comprehensible.

The extracted temporal envelope 1728 is used to activate the cutaneousactuators 1722 using the carrier signals 1734. The carrier signals 1734have a periodic waveform (e.g., sinusoidal) that is altered usingfeatures of the temporal envelope 1728 for the purpose of conveying thespeech information via the actuator signals 1718. Features of thetemporal envelope 1728 may include amplitude, frequency, phase,periodicity, wavelength, etc. In one embodiment shown in FIG. 17B, thecarrier signals 1734 may be sinusoidal signals describing a smoothrepetitive oscillation. In other embodiments, the carrier signals 1734may be a pulse wave or pulse train, any non-sinusoidal periodicwaveform, or a square wave train having a regular duty cycle. Thecarrier signals 1734 typically have a higher frequency than the temporalenvelope 1728. In one embodiment, the cutaneous actuators 1722 areoscillated using carrier signals 1734 having a periodic (e.g.,sinusoidal or pulse wave train waveform) whose frequency is advantageousfor human perception, e.g., between 10 Hz to 300 Hz. The carrier signals1734 may be altered using frequency modulation (FM) or amplitudemodulation (AM) to generate the actuator signals 1718.

FIG. 17C is an illustration of example encoded speech signals 1706 fortransmission to the cutaneous actuators 1722, in accordance with anembodiment. The signals shown in FIG. 17C include the actuator signals1736 having varying amplitude (e.g., 1738 and 1740) and actuator signals1742 having varying frequency (e.g., 1744 and 1746). The actuatorsignals 1736 may be generated using amplitude changing methods totransmit information related to the temporal envelope 1728 over thecontinuous-wave carrier signals 1734. The amplitude (signal strength) ofthe carrier signals 1734 is varied in proportion to the waveform of thetemporal envelope 1728 being transmitted to generate the actuatorsignals 1736.

The actuator signals 1742 may be generated using frequency alteringmethods to transmit information related to the temporal envelope 1728over the continuous-wave carrier signals 1734. As shown in FIG. 17C, Theactuator signals 1742 may be generated by encoding information from thetemporal envelope 1728 in the carrier signals 1734 by varying thefrequency of the carrier signals 1734. The frequency deviation (thedifference between the frequency of the carrier signals 1734 and thecenter frequency of the actuator signals 1742 may be proportional to thetemporal envelope 1728.

FIG. 17D is a block diagram of an example envelope encoder 1716 forencoding of speech signals 1706 and transmission to cutaneous actuators1722, in accordance with an embodiment. FIG. 17D includes a rectifier1748, a low pass filter 1752, a high pass filter 1758, a machinelearning engine 1760, a carrier signal generator 1772, and an actuatorsignal generator 1776. In other embodiments, the envelope encoder 1716may include additional or fewer components than those described herein.Similarly, the functions can be distributed among the components and/ordifferent entities in a different manner than is described here.

The envelope encoder 1716 may be implemented in software, hardware, or acombination thereof. For example, the envelope encoder 1716 may be partof a PC, a tablet PC, an STB, a smartphone, an IoT appliance, or anymachine capable of executing instructions that specify actions to betaken by that machine. The envelope encoder 1716 may include one or moreprocessing units (e.g., a CPU, a GPU, a DSP, a controller, a statemachine, one or more ASICs, one or more RFICs, or any combination ofthese) and a memory.

The rectifier 1748 converts alternating signals, such as the speechsignals 1706 shown in FIG. 17B, which periodically reverse direction, todirect signals that flow in only one direction. In one embodiment, therectifier 1748 may be made of semiconductor diodes, silicon-controlledrectifiers, or other silicon-based semiconductor switches. In oneembodiment, the rectifier 1748 may be implemented as part of a DSP or insoftware. The rectifier 1748 generates the rectified signals 1750 byrectifying the speech signals 1706. Because of the alternating nature ofthe input speech signals 1706, the rectifier 1748 may produce DCrectified signals 1750 that, though unidirectional, include pulses. Therectified signals 1750 may be smoothed by an electronic filter, whichmay be a set of capacitors or chokes, followed by a voltage regulator.

As shown in FIG. 17D, the low pass filter 1752 transmits the lowfrequency components 1756 (components with a frequency lower than acertain threshold frequency) of the rectified signals 1750 to theactuator signal generator 1776 and attenuates (filters out) highfrequency components of the rectified signals 1750 with frequencieshigher than the threshold frequency. The low pass filter 1752 may beimplemented in the form of a hiss filter, an anti-aliasing filter forconditioning signals prior to analog-to-digital conversion, or a digitalfilter. In one embodiment, the low pass filter 1752 generates thetemporal envelope 1754 of the speech signals 1706 by filtering outfiltering out frequency components of the rectified signals 1750 thatare higher than a threshold frequency. By filtering out these highfrequency components, the remaining signals consist of slowly varyingcomponents that track the temporal envelope 1754 of the rectifiedsignals 1750. As shown in FIG. 17D, the low pass filter 1752 may alsodirectly receive the speech signals 1706 and transmit low frequencycomponents 1756 of the speech signals 1706 to the actuator signalgenerator 1776 by filtering out filtering out frequency components ofthe speech signals 1706 that are higher than a threshold frequency.These low frequency components 1756 of the speech signals 1706 are usedby the actuator signal generator 1776 to determine a low frequencysignal power of the low frequency components 1756 of the speech signals1706 and generate the actuator signals 1718, as described below withrespect to the actuator signal generator 1776.

As shown in FIG. 17D, the high pass filter 1758 receives the speechsignals 1706 and transmits the high frequency components 1768(components with a frequency higher than a certain threshold frequency)of the speech signals 1706 to the actuator signal generator 1776 andattenuates (filters out) the frequency components of the speech signals1706 with frequencies lower than the threshold frequency. The high passfilter 1752 may be implemented in the form of a first-order electronichigh-pass filter by placing the input speech signals 1706 across aseries combination of a capacitor and a resistor for conditioningsignals prior to analog-to-digital conversion, or as a discrete-timewashout filter, or a digital filter. The high frequency components 1768of the speech signals 1706 are used by the actuator signal generator1776 to determine a high frequency signal power of the high frequencycomponents 1768 of the speech signals 1706 and generate the actuatorsignals 1718, as described below with respect to the actuator signalgenerator 1776.

The carrier signal generator 1772 generates and transmits periodicpulses or waveform carrier signals 1774 at a steady base frequency ofalternation to the actuator signal generator 1776 on which informationfrom the temporal envelope 1754 can be imposed by the actuator signalgenerator 1776 by increasing signal strength, varying the basefrequency, varying the wave phase, or other means. In one embodiment,the carrier signal generator 1772 may include a reference oscillator tosupply a periodic wave with a known frequency to a phase-locked loop(PLL), which generates the carrier signals 1774 at a desired frequency(e.g., 10-300 Hz) and a level controller to determine the amplituderange of the carrier signals 1774.

As shown in FIG. 17D, the machine learning engine 1760 receives thespeech signals 1706 and generates haptic cues 1770, which are used bythe actuator signal generator 1776 to generate the actuator signals1718. The machine learning engine 1760 may be implemented in software,hardware, or a combination thereof. For example, the machine learningengine 1760 may include software modules, one or more processing units(e.g., a CPU, a GPU, a DSP, a controller, a state machine, one or moreASICs, or any combination of these) and a memory. The operation ofmachine learning engine 1760 is illustrated and described in detailabove with reference to FIGS. 4 and 13A-13D.

The machine learning engine 1760 includes a feature extractor 1762 and amachine learning model 1766. The feature extractor 1762 extracts afeature vector 1764 from the speech signals 1706 and transmits thefeature vector 1764 to the machine learning model 1766 as shown in FIG.17D. The feature extractor 1762 may be implemented in software,hardware, or a combination thereof. The feature vector 1764 mayrepresent amplitudes of frequency bands of the speech signals 1706.These frequency bands may be obtained by decomposing the speech signals1706 into the frequency bands by the feature extractor 1762, the machinelearning engine 1760, or another component of the envelope encoder 1716.For example, an array of band-pass filters that span the speech spectrummay be used to decompose the speech signals 1706 into the frequencybands. The operation of the feature extractor 1762 and the featurevector 1764 are illustrated and described in detail above with referenceto FIGS. 4 and 13A-13D.

The machine learning engine 1760 may use supervised machine learning totrain the machine learning model 1766 with feature vectors from apositive training set and a negative training set serving as the inputs.Different machine learning techniques—such as linear support vectormachine (linear SVM), boosting for other algorithms (e.g., AdaBoost),neural networks, logistic regression, naïve Bayes, memory-basedlearning, random forests, bagged trees, decision trees, boosted trees,or boosted stumps—may be used in different embodiments.

The machine learning model 1766 generates haptic cues 1770 using thefeature vector 1764. Each of these haptic cues 1770 may indicate thehaptic output for a cutaneous actuator 1722 and may be represented byone or more nodes in a final layer of the machine learning model 1766.For example, each node of the machine learning model 1766 may indicatefor each cutaneous actuator 1722, a percentage value that may be used todetermine whether or not to activate a particular state of the cutaneousactuator 1722.

The machine learning model 1766 may use a cost function based on areconstruction error and constraints. The cost function is an evaluationfunction that measures how well the machine learning model 1766 maps thetarget haptic cues 1770. In addition, the cost function may determinehow well the machine learning model 1766 performs in supervisedprediction or unsupervised optimization. The constraints cause thesequence of haptic cues 1770 that are output to have highcompressibility, low entropy, low ordinality (i.e., fewer number ofpossible haptic outputs), sparsity, temporal stability, and spatialstability. The actuator signal generator 1776 generates the actuatorsignals 1718 by encoding the haptic cues 1770 into the carrier signals1774 as described below with respect to the actuator signal generator1776. The operation of the machine learning model 1766, haptic cues1770, cost function, reconstruction error, and constraints areillustrated and described in detail above with reference to FIGS. 4 and13A-13D.

The actuator signal generator 1776 generates the actuator signals 1718by encoding features of the temporal envelope 1754 of the speech signals1706, low frequency components 1756 of the speech signals 1706, highfrequency components 1758 of the speech signals 1706, and the hapticcues 1770 into the carrier signals 1774. The actuator signal generator1776 may be implemented in software, hardware, or a combination thereof.For example, the actuator signal generator 1776 may include one or moreprocessing units (e.g., a CPU, a GPU, a DSP, a controller, a statemachine, one or more ASICs, one or more RFICs, or any combination ofthese) and a memory.

In one embodiment, the actuator signal generator 1776 extracts thetemporal envelope 1754 directly from the speech signals 1706 byperforming a Hilbert transformation on the speech signals. The Hilberttransform is a specific linear operator that takes the speech signals1706 and produces the analytic representation of the speech signals1706. The actuator signal generator 1776 extracts the temporal envelope1754 from the magnitude (modulus) of the analytic representation of thespeech signals 1706.

In one embodiment, the actuator signal generator 1776 decomposes thespeech signals 1706 into frequency bands. For example, an array ofband-pass filters that span the speech spectrum may be used to decomposethe speech signals 1706 into the frequency bands. The actuator signalgenerator 1776 extracts a corresponding temporal envelope (“ENVi”) fromeach frequency band Fi. Each corresponding temporal envelope ENVirepresents changes in amplitude (e.g., 1732) of the frequency band Fi.The actuator signal generator 1776 generates the actuator signals 1718by encoding changes in the amplitude of the frequency band Fi from eachcorresponding temporal envelope ENVi into the carrier signals 1774.

In one embodiment, the actuator signal generator 1776 generates theactuator signals (e.g., signals 1736 in FIG. 17C) by determining theamplitude (e.g., 1732) of the speech signals from the temporal envelope1728. For example, the actuator signal generator 1776 may determine theamplitude by determining the magnitude of the analytic signal associatedwith the temporal envelope 1728. To encode features of the temporalenvelope 1728 into the carrier signals 1774, actuator signal generator1776 increases or decreases the amplitude of the carrier signals 1774proportional to the amplitude of the speech signals 1706. In oneexample, features of the temporal envelope 1728 are superimposed ontothe amplitude of the carrier signals 1774. The carrier signals 1774 arealtered such that the amplitude of the carrier signals 1774 varies inline with the variations in intensity of the speech signals 1706. Inthis way, the amplitude of the carrier signals 1774 carry the featuresof the temporal envelope 1728. The altered carrier signals 1774 aretransmitted as the actuator signals 1736 to the cutaneous actuators1722.

In one embodiment, the actuator signal generator 1776 encodes featuresof the temporal envelope 1728 into the carrier signals 1774 byincreasing or decreases the frequency of the carrier signals 1774proportional to the amplitude (e.g., 1732) of the speech signals 1706.The carrier signals 1774 are altered such that the frequency of thecarrier signals 1774 varies in line with the variations in intensity ofthe speech signals 1706. When features of the temporal envelope 1728 areencoded into the carrier signals 1774 to generate the actuator signals(e.g., signals 1743 in FIG. 17C), the actuator signals 1743 increase anddecrease in frequency proportional to the amplitude of the speechsignals 1706. The amount by which the actuator signals 1743 increase anddecrease in frequency is known as the deviation. In this way, thefrequencies of the carrier signals 1774 carry the features of thetemporal envelope 1728. The altered carrier signals 1774 are transmittedas actuator signals (e.g., signals 1743 in FIG. 17C) to the cutaneousactuators 1722.

In one embodiment, the actuator signal generator 1776 encodes featuresof the temporal envelope 1728 into the carrier signals 1774 byincreasing or decreasing a number of the cutaneous actuators 1722activated proportional to the amplitude (e.g., 1732) of the speechsignals. For example, as the amplitude 1732 according to the temporalenvelope 1728 increases, the carrier signals 1774 are altered such thatthe actuator signals (e.g., 1718) activate (turn on) a larger number ofthe cutaneous actuators 1722 on the haptic communication device 1720.This may result in a larger area of the user's body 1726 receivingstimulation from the haptic vibrations 1724. This may also create asense of continuous tactile motion (e.g., stroking or moving touch) onthe body 1726 rather than discrete actuation points, as described abovewith reference to FIG. 15B. As the amplitude 1732 according to thetemporal envelope 1728 decreases, the carrier signals 1774 are alteredsuch that the actuator signals 1718 activate a smaller number of thecutaneous actuators 1722 on the haptic communication device 1720.Altering the number of cutaneous actuators 1722 activated results inmore sophisticated haptic communication effects. The information relatedto the words of a social touch lexicon described above with reference toFIG. 15B may thus communicated by altering the number of the cutaneousactuators 1722 activated proportional to the amplitude 1732 of thespeech signals. By changing the amplitude and patterns of vibration, alarge number of combinations, rhythms or messages may be reproduced.

In one embodiment, the actuator signal generator 1776 encodes the amountof high-frequency signal power (which is present in fricatives such as/s/, /f/, and /sh/) into the carrier signals 1774 to generate theactuator signals 1718. The distribution of signal power may bedetermined using the low-pass filter 1752 and the high-pass filter 1758.The cutoff frequencies of the filters may be set to an intermediateregion useful for distinguishing the presence or absence ofhigh-frequency speech, e.g., 4 kHz. The ratio of the power output ofeach filter may be used to alter the carrier signals 1774 to generatethe actuator signals 1718 to change the activity of cutaneous actuators1722. Similarly, the presence or the absence of formant structure in thespeech signals 1706 may be encoded into the carrier signals 1774 togenerate the actuator signals 1718. Similarly, the particular formantsthat are present in the speech signals 1706 may be encoded into thecarrier signals 1774 to generate the actuator signals 1718.

In one embodiment, the compressed spectrum of the speech signals 1706may be divided among the cutaneous actuators 1722. For example, vowelscorrespond to low frequency energy and the localization of the energy inmultiple discrete energy bands (‘formants’) as illustrated above in FIG.7. As can be seen from the spectrogram the vowels ‘A’ and ‘oo’ havemultiple bands of relatively high energy in the low frequency region. Incontrast ‘f’, ‘c’, ‘b’, and ‘k’ are brief, low energy, and lack thelocalized energy band structure that vowels have. The actuator signalgenerator 1776 may use the detection of low energy signals that lack thefrequency structure of vowels to generate the actuator signals 1718,allowing the user more time to appreciate the subtler pattern of theconsonant. In one embodiment, the actuator signals 1718 may be generatedto transmit the haptic vibrations 1724 corresponding to vowels of thereceived speech signals 1706 at a first speed. The cutaneous actuators1722 may transmit haptic vibrations 1724 corresponding to consonants ata second speed lower than the first speed.

In one embodiment, the actuator signal generator 1776 determines a lowfrequency signal power of the low frequency components 1756 of thespeech signals 1706. The actuator signal generator 1776 may determinethe low frequency signal power as a sum of the absolute squares oftime-domain samples divided by the signal length. The actuator signalgenerator 1776 similarly determines a high frequency signal power of thehigh frequency components 1768 of the speech signals 1706. The actuatorsignal generator 1776 determines a ratio of the high frequency signalpower to the low frequency signal power. The actuator signal generator1776 generates the actuator signals 1718 by encoding the ratio of thehigh frequency signal power to the low frequency signal power into thecarrier signals 1774 as described above. For example, the actuatorsignal generator 1776 may perform the encoding of the carrier signals1774 by increasing or decreases the amplitude or frequency of thecarrier signals 1774 proportional to the ratio of the high frequencysignal power to the low frequency signal power. Encoding the powercharacteristics of the spectral bands of the speech signals 1706 enablesthe actuator signals 1718 to capture a more complete description of thelinguistic units.

In one embodiment, the actuator signal generator 1776 encodes the hapticcues 1770 into the carrier signals 1774 by altering the frequency of thecarrier signals 1774 or altering a number of the cutaneous actuators1722 activated based on the haptic cues 1770. The actuator signalgenerator 1776 may encode the haptic cues 1770 into the carrier signals1774 using any of the methods described above with respect to FIGS.13A-13D and 17D.

In alternative embodiments of the envelope encoder 1716, the carriersignal generator 1772 may be located within the actuator signalgenerator 1776. In these embodiments, the temporal envelope 1754 may betransmitted by the actuator signal generator 1776 to the PLL section ofthe carrier signal generator 1772 to alter the frequency values of thegenerated carrier signals 1774 to generate the actuator signals 1718.The temporal envelope 1754 may be transmitted by the actuator signalgenerator 1776 to the level controller section of the carrier signalgenerator 1772 to alter the amplitude values of the generated carriersignals 1774 to generate the actuator signals 1718.

The benefits and advantages of the embodiments disclosed herein are asfollows. The temporal envelope is a one-dimensional signal and may berepresented as a single time-varying waveform to reduce memory andprocessing requirements and increase efficiency of transmission.Therefore, the disclosed embodiments lead to efficiency in memory,processing, and transmission over frequency decomposition methods. Thedisclosed embodiments further lead to a low-dimensional approach tohaptic encoding. The processing power needed is reduced due to thesmaller number of channels that are processed and because temporalenvelope extraction is performed with less computational complexity thanfrequency decomposition. Additional advantages and benefits of thedisclosed embodiments (especially related to using the ratio of the highfrequency signal power to the low frequency signal power to driveactuator activity) are that vowels are relatively easy to recognizeusing automated techniques and by humans in a haptic representation ofspeech. Consonants are shorter in duration and have ambiguous spectralsignatures. By slowing the dynamics of the haptic presentation duringconsonants, relative to the speed at which vowels are displayed, theaccuracy of perception will increase with less degradation of speedcompared to uniform slowing.

FIG. 17E is an illustration of an example process for envelope encodingof speech signals 1706 and transmission to cutaneous actuators 1722, inaccordance with an embodiment. In one embodiment, the process of FIG.17E is performed by the haptic communication system 1700. Other entities(e.g., a remote console or computer) may perform some or all of thesteps of the process in other embodiments. Likewise, embodiments mayinclude different and/or additional steps, or perform the steps indifferent orders.

The haptic communication system 1700 receives 1780 speech sounds 1702 ora textual message 1710. The speech sounds 1702 may be received by themicrophone 1704. The textual message 1710 may be received by the speechsynthesizer 1712.

The haptic communication system 1700 generates 1782 speech signals(e.g., 1706 or 1714) corresponding to the speech sounds 1702 or thetextual message 1710. The microphone 1704 may generate the speechsignals 1706 by digitizing the speech sounds 1702. The speechsynthesizer 1712 may generate the speech signals 1706 by mapping thetextual message 1710 to stored sound signals.

The haptic communication system 1700 extracts 1784 a temporal envelope1754 from the speech signals 1706. The temporal envelope 1754 representschanges in amplitude of the speech signals 1706.

The haptic communication system 1700 generates 1786 carrier signals 1774having a periodic waveform. The carrier signal generator 1772 maygenerate periodic pulses or waveform carrier signals 1774 at a steadybase frequency of alternation on which information from the temporalenvelope 1754 can be imposed.

The haptic communication system 1700 generates 1788 the actuator signals1718 by encoding features of the temporal envelope 1728 and the speechsignals 1706 into the carrier signals 1774. For example, the actuatorsignal generator 1776 may encode features of the temporal envelope 1728into the carrier signals 1774 by increasing or decreases the frequencyor amplitude of the carrier signals 1774 proportional to the amplitude(e.g., 1732) of the speech signals 1706.

The haptic communication system 1700 generates 1790, by the cutaneousactuators 1722, haptic vibrations 1724 representing the speech sounds1702 or the textual message 1710 using the actuator signals 1718. Speechmessages may therefore be transmitted from one user to another user andconverted to haptic messages for greeting, parting, giving attention,helping, consoling, calming, pleasant, and reassuring touches.

Haptic Communication System Using Broadband Actuator Signals forTransmission to Cutaneous Actuators

Embodiments also relate to a haptic communication system including abroadband signal generator. The broadband signal generator receivessensor signals from a sensor signal generator such as a haptic sensor ora microphone. The sensor signals describe a message, such as a hapticsocial touch or speech for transmission to a user. Parameters describingthe message are extracted from the sensor signals. In one embodiment,broadband carrier signals having a large number of frequencies may begenerated by aggregating frequency components. Actuator signals meantfor transmission to a haptic communication device are generated byencoding the parameters from the sensor signals into the broadbandcarrier signals. Cutaneous actuators are embedded in the hapticcommunication device, which is communicatively coupled to the broadbandsignal generator to receive the actuator signals. Haptic vibrations aregenerated by the cutaneous actuators corresponding to the actuatorsignals on a body of the user to communicate the message to the user.

FIG. 18A is a block diagram illustrating an example haptic communicationsystem 1800 using broadband actuator signals 1818 for transmission tocutaneous actuators 1822, in accordance with an embodiment. The hapticcommunication system 1800 includes a sensor signal generator 1804, abroadband signal generator 1816, and a haptic communication device 1820.In other embodiments, the haptic communication system 1800 may includeadditional or fewer components than those described herein. Similarly,the functions can be distributed among the components and/or differententities in a different manner than is described here.

As shown in FIG. 18A, the haptic communication system 1800 includes asensor signal generator 1804 that receives a message from a sending userand generates sensor signals 1806 corresponding to the message. In oneembodiment, the sensor signal generator 1804 may be a microphone thatreceives speech sounds 1802 uttered by the sending user and generatesspeech signals corresponding to the speech sounds. In anotherembodiment, the sensor signal generator 1804 may be a speech synthesizerthat receives a textual message typed out by the sending user andgenerates speech signals corresponding to the textual message. Inanother embodiment, the sensor signal generator 1804 may be a hapticsensor that receives haptic input (e.g., a social touch) from thesending user and generates sensor signals 1806 corresponding to thehaptic input. The sensor signals 1806 may be analog signals, digitalsignals, or a combination thereof. In addition, the sensor signalgenerator 1804 may include an analog-to-digital converter to digitizethe message to the sensor signals 1806. The sensor signal generator 1804is communicatively coupled to and transmits the sensor signals 1806 tothe broadband signal generator 1816 over a wired or a wirelessconnection. Portions of the sensor signal generator 1804 may beimplemented in software or hardware. For example, the sensor signalgenerator 1804 may be part of a smartphone, an internet of things (IoT)appliance, or any machine capable of executing instructions that specifyactions to be taken by that machine.

The broadband signal generator 1816 (illustrated and described in moredetail below with respect to FIG. 18C) encodes parameters of the sensorsignals 1806 to generate the actuator signals 1818. In one embodiment,the broadband signal generator 1816 extracts parameters from the sensorsignals 1806 describing the message for transmission to a body 1826 of auser. The parameters may include changes in amplitude or frequency ofthe sensor signals 1806 over time or parameters of haptic input, such aspressure, temperature, sheer stress, duration, and space dimensions of ahaptic touch.

The broadband signal generator 1816 generates broadband carrier signals(e.g., the broadband carrier signals 1828 shown below in FIG. 18B) byaggregating frequency components. The broadband signal generator 1816generates the actuator signals 1818 by encoding the parameters from thesensor signals 1806 into the broadband carrier signals 1828. Thebandwidth of the actuator signals 1818 is limited to the range of theresponsiveness of the mechanical receptors of the body 1826, between 10and 300 Hz. The actuator signals 1818 thus carry information from thesensor signals 1806 encoded within the broadband carrier signals 1828.Portions of the broadband signal generator 1816 may be implemented insoftware or hardware. For example, the broadband signal generator 1816may be part of a PC, a tablet PC, an STB, a smartphone, an internet ofthings (IoT) appliance, or any machine capable of executing instructionsthat specify actions to be taken by that machine.

As shown in FIG. 18A, the haptic communication device 1820 receives theactuator signals 1818 and generates and transmits haptic vibrations 1824to transmit the message to the body 1826 of the user wearing the hapticcommunication device 1820. The haptic communication device 1820 mayinclude an array of cutaneous actuators 1822 that generates the hapticvibrations 1824. The cutaneous actuators 1822 are communicativelycoupled over wired or wireless connections to the broadband signalgenerator 1816 to generate the haptic vibrations 1824 based on theactuator signals 1818. The haptic communication device 1820 may includeone or more processing units (e.g., a central processing unit (CPU), agraphics processing unit (GPU), a digital signal processor (DSP), acontroller, a state machine, one or more application specific integratedcircuits (ASICs), one or more radio-frequency integrated circuits(RFICs), or any combination of these) and a memory.

In one embodiment, the cutaneous actuators 1822 may be voice coils thatconvert an analog AC-coupled signal (the actuator signals 1818) toproportional mechanical motion. The cutaneous actuators 1822 may bearranged on a semi-rigid backing that secures them in position againstthe body 1826 with enough pressure that the haptic vibrations 1824 canbe perceived by the user. In other embodiments, the haptic communicationdevice 1820 may include piezoelectric, electroactive polymer, eccentricrotating mass, and linear resonant actuators. Other embodiments of thehaptic communication device 1820 are illustrated and described in detailabove with respect to FIGS. 1, 8B, 8E, 9A-9E, 11A, 11I, 12A-12B, 15A and16A-16L among others.

FIG. 18B is waveform diagrams illustrating example broadband carriersignals 1828, in accordance with an embodiment. The waveforms shown inFIG. 18B include the broadband carrier signals 1828 and a power spectrum1862 of the broadband carrier signals 1828.

As shown in FIG. 18B, the broadband carrier signals 1828 contain manyfrequency components including a wide range of frequencies that can betransmitted, recorded, and manipulated. In one embodiment, the broadbandcarrier signals 1828 may include a number of frequency components havingnearly equal power or amplitude at different frequencies, thus givingthe broadband carrier signals 1828 a uniform power spectrum 1862. In oneembodiment, the broadband carrier signals 1828 may be constructed from anumber of discrete samples that are a sequence of serially uncorrelatedrandom variables with zero mean and finite variance. The samples mayalso be independent and have identical probability distributions. In oneembodiment, the broadband carrier signals 1828 may contain manyfrequency components having a power spectrum 1862 such that the powerspectral density (energy or power per frequency interval) is inverselyproportional to frequency of the frequency component.

The power spectrum 1862 of the broadband carrier signals 1828illustrates the distribution of signal power among the differentfrequency components composing the broadband carrier signals 1828. For asinusoid, the signal power is usually concentrated at the frequency ofthe sinusoid. For the broadband carrier signals 1828, however, thesignal power is spread over a range of frequencies. The power spectrum1862 of the broadband carrier signals 1828 is wide compared to that of asingle sinusoid or sums of a few sinusoids. The peaks (e.g., 1832) ofthe power spectrum 1862 of the broadband carrier signals 1828 are moreuniform. For example, the difference between peak 1832 at frequency 1830and peak 1836 at frequency 1834 is less than a threshold power value.Similarly, the difference between peak 1832 at frequency 1830 and peak1840 at frequency 1838 is less than the threshold power value.

The broadband carrier signals 1828 may be generated in a variety ofways, as described in detail below with respect to FIG. 18C. In oneembodiment, a desired spectrum of frequencies (e.g., 10 Hz to 300 Hz) isdefined. The frequency components are transformed from the frequencydomain representation of a flat power spectrum to the time domain bytaking the inverse Fourier transform. In another embodiment, a noiseprocess is simulated by aggregating a large number of sinusoids (e.g.,10-1000 sinusoids). A continuum of frequencies is approximated by addingsuccessive frequencies with a small step size, e.g., 1-10 Hz. In anotherembodiment, a smaller number of sinusoids (e.g., 3-10) are aggregatedthat span the frequency range of responsiveness of the skin. In anotherembodiment, the broadband carrier signals 1828 may be constructed in thetime domain by taking each time sample of the broadband carrier signals1828 as being a realization of a random number, e.g., a Gaussiandistributed random variable that is generated independently of thevalues that preceded it. Such a method would result in a white Gaussiannoise process. In another embodiment, periodic, low-frequency,non-sinusoidal signals may be used to generate the broadband carriersignals 1828, since such signals have frequency spectra that arecomprised of a fundamental frequency and harmonic frequencies.

The broadband signal generator 1816 may generate the actuator signals1818 by encoding the parameters from the sensor signals 1806 into thebroadband carrier signals 1828 in a number of ways. In one embodiment,the physical location on the body 1826 of the user that is beingactuated may be encoded into the broadband carrier signals 1828. Forexample, the location of the particular one or more actuators that aredriven by the actuator signals 1818 convey information (e.g.,identifying a particular phoneme) to the body 1826 of the user. Inanother embodiment, the amplitude of each frequency component of speechsignals or the sensor signals 1806 can be used to encode information. Inanother embodiment, sequences of activation of different actuators canbe used to encode information. In another embodiment, features of thepower spectrum 1862 can be used. For example, actuator signals 1818whose power is concentrated in the lower frequencies can conveydifferent information than signals whose power is concentrated at higherfrequencies. Non-linguistic applications are possible as well. Forexample, in simulated social touch using the haptic communication device1820, broadband actuator signals 1818 convey a stimulus that is morenatural than sinusoidal stimuli, which have an intense, artificial,mechanical sensation.

FIG. 18C is a block diagram of an example broadband signal generator1816, in accordance with an embodiment. FIG. 18C includes a parameterextractor 1842, a carrier signal generator 1872 and an actuator signalgenerator 1876. In other embodiments, the broadband signal generator1816 may include additional or fewer components than those describedherein. Similarly, the functions can be distributed among the componentsand/or different entities in a different manner than is described here.

Portions of the broadband signal generator 1816 may be implemented insoftware, hardware, or a combination thereof. For example, the broadbandsignal generator 1816 may be part of a PC, a tablet PC, an STB, asmartphone, an IoT appliance, or any machine capable of executinginstructions that specify actions to be taken by that machine. Thebroadband signal generator 1816 may include one or more processing units(e.g., a CPU, a GPU, a DSP, a controller, a state machine, one or moreASICs, one or more RFICs, or any combination of these) and a memory.

The parameter extractor 1842 extracts parameters 1852 from the sensorsignals 1806 describing the message for transmission to the body 1826 ofthe user. The parameter extractor 1842 includes band pass filters 1844and an amplitude detector 1848. In other embodiments, the parameterextractor 1842 may include additional or fewer components than thosedescribed herein. Similarly, the functions can be distributed among thecomponents and/or different entities in a different manner than isdescribed here.

The parameter extractor 1842 may extract parameters 1852 related tosocial touch, such as forces, vibrations, or motions. The parameters1852 may be represented in dimensions of pressure, temperature, texture,sheer stress, time, and space or a subset thereof. These parameters 1852(e.g., pressure) may be used by the haptic communication device 1820 togenerate haptic vibrations 1824 using the parameters 1852. For example,the parameters 1852 may be translated into a duration time, frequency,or amplitude of the haptic vibrations 1824.

In one embodiment, the parameter extractor 1842 may determine, from thesensor signals 1806, which of the cutaneous actuators 1822 should beturned ON (i.e., activated) or OFF at different times to createdifferent haptic illusions or convey different messages to the body 1826of the user, as described and illustrated above with respect to FIG.15B.

In one embodiment, the parameter extractor 1842 may split the sensorsignals 1806 into speech subcomponents, as described and illustratedabove with respect to FIG. 3. The speech subcomponents may include oneor more of phonemes of the received sensor (speech) signals 1806,frequencies of the received sensor signals 1806, formants of thereceived sensor signals 1806, and semantics of the received sensorsignals 1806. A phoneme is any of the perceptually distinct units ofsound in a specified language that distinguish one word from another,for example p, b, d, and t in the English words pad, pat, bad, and bat.A formant refers to each of several prominent bands of frequency thatdetermine the phonetic quality of vowels in the sensor signals 1806.Semantics may include logical aspects of meaning, such as sense,reference, implication, and logical form, lexical semantics (wordrelations), or conceptual semantics (the cognitive structure ofmeaning). The parameters related to speech phonemes extracted from thesensor signals 1806 may be used to create different arrangements ofactivated cutaneous actuators 1822 and different haptic patterns on thebody 1826, wherein each pattern may be associated with a particularphoneme.

The band pass filters 1844 decompose the sensor signals 1806 intofrequency bands 1846. For example, an array of band-pass filters 1844that span the speech spectrum may be used to decompose the sensorsignals 1806 into the frequency bands 1846. Each band-pass filter passesfrequencies within a certain range and rejects (attenuates) frequenciesoutside that range. The band pass filters 1844 may be constructed usingan analog resistor-inductor-capacitor circuit or digital filters thatperform mathematical operations on the sampled sensor signals 1806. Forexample, an analog-to-digital converter may be used to sample the sensorsignals 1806, followed by a microprocessor and peripheral componentssuch as memory to store data and filter coefficients, etc. Finally adigital-to-analog converter completes the output stage.

The amplitude detector 1848 detects an amplitude of each frequency band1846. The amplitude detector 1848 may be an electronic circuit thattakes the frequency bands 1846 as input and provides an output which isthe envelope of each frequency band 1846. For example, the amplitudedetector 1848 may include a level controller to determine the amplituderange of each frequency band. The amplitude detector 1848 may include acapacitor to store up charge on the rising edges of each frequency bandsignal, and release it through a resistor when the signal falls. Theamplitudes and frequency bands 1846 make up the parameters 1852 used togenerate the actuator signals 1818.

The carrier signal generator 1872 generates the broadband carriersignals 1828 by aggregating a plurality of frequency components. Thecarrier signal generator 1872 includes a frequency generator 1856 and arandom amplitude generator 1860. In other embodiments, the carriersignal generator 1872 may include additional or fewer components thanthose described herein. Similarly, the functions can be distributedamong the components and/or different entities in a different mannerthan is described here.

The frequency generator 1856 generates the frequency components 1858 foruse by the actuator signal generator 1876. The frequency generator 1856may generate periodic pulses or waveforms at different frequencies ofalternation. In one embodiment, the frequency generator 1856 may includea reference oscillator to supply a periodic wave with a known frequencyto a phase-locked loop (PLL), which generates each frequency component1858 at a desired frequency (e.g., 10-300 Hz). In one embodiment, eachof the frequency components (e.g., fi) has a sinusoidal waveform and adifference in frequency (fj−fi) between each pair fi, fj of adjacentfrequency components is less than a threshold frequency ft. For example,the frequency components may be sinusoidal signals having frequencies 10Hz, 10.2 Hz, 10.4 Hz, . . . , 300 Hz. Here the difference in frequency(e.g., 10.4 Hz-10.2 Hz) between each pair fi, fj of adjacent frequencycomponents is less than a threshold frequency (e.g., 0.29 Hz). In oneembodiment, a number of the frequency components is between 10 and 1000and each of the frequency components has a frequency between 10 Hz and300 Hz. For example, if there are 1000 evenly spaced frequencycomponents between 10 Hz and 300 Hz, the difference in frequency betweenadjacent frequency components will be (300−10)=290/1000=0.29 Hz.

In one embodiment, a difference (a2−a1) between an amplitude a1 of afirst frequency component having a highest amplitude and an amplitude a2of a second frequency component having a lowest amplitude is less than athreshold amplitude aT. Similar to the power spectrum 1862 illustratedand described above with respect to FIG. 18B, the amplitude of thecarrier signals 1828 generated is a random signal having nearly equalamplitude at different frequencies, giving it a nearly flat amplitudespectrum.

In one embodiment, an amplitude of each frequency component fi having afrequency less than (fi<ft) a threshold frequency ft is greater than anamplitude of each frequency component fj having a frequency greater than(fj>ft) the threshold frequency ft. For example, the broadband carriersignals 1828 generated by aggregating such frequency components willhave a power spectrum 1862 such that the power spectral density that maybe inversely proportional to frequency of the frequency component fi.

In one embodiment, an amplitude of each frequency component fi decreasesas a frequency of the frequency component fi increases and the amplitudeof the frequency component fi increases as the frequency of thefrequency component fi decreases. For example, the power per Hz for suchfrequency components decreases as the frequency increases. In anotherexample, the power per octave may be equal. An octave is a frequencycomponent whose highest frequency is twice its lowest frequency. Forexample, the band from 20 Hz to 40 Hz is an octave, as is the band from40 to 80 Hz. Thus, although the power per Hz decreases with increasingfrequency, the width of successive octaves increases (they contain morefrequencies), giving the frequency components equal power per octave.

In one embodiment, the frequency components have periodic,low-frequency, non-sinusoidal waveforms. Each of the frequencycomponents has a fundamental frequency fi less than a thresholdfrequency ft. The frequency component also has harmonic frequencies ofthe fundamental frequency fi. For example, the fundamental frequency fimay be kept low (e.g., 10 Hz) and the harmonic frequencies included togenerate the broadband carrier signals 1828. Each harmonic of thefundamental frequency fi has a waveform with a frequency that is apositive integer multiple of the fundamental frequency fi. The harmonicsare periodic at the fundamental frequency; the sum of harmonics is alsoperiodic at that frequency. For example, if the fundamental frequency is10 Hz, the frequencies of the first three higher harmonics are 20 Hz(2nd harmonic), 30 Hz (3rd harmonic), and 30 Hz (4th harmonic).

In one embodiment, the random amplitude generator 1860 may generate thebroadband carrier signals 1828 by randomly assigning an amplitude toeach of the frequency components. For example, the random amplitudegenerator 1860 may generate broadband carrier signals 1828 that are Tseconds in duration and use a sampling rate of Fs samples/second. Therandom amplitude generator 1860 forms a sequence x[n] from N randomnumbers for the frequency components, where N=T×Fs. The random amplitudegenerator 1860 determines the average value of the sequence, mu=ave(x).The random amplitude generator 1860 subtracts the average value from thesequence, y=x−mu. The random amplitude generator 1860 identifies themaximum excursion, m=max(|y|) (i.e., the maximum of the absolute valueof y). The random amplitude generator 1860 scales the sequence by themaximum excursion to generate the broadband carrier signals 1828, suchthat the scaled sequence z has a maximum excursion of 1, z=y/m. Theresulting broadband carrier signals 1828 may be transmitted to adigital-to-analog converter using a sample rate of Fs. The analog outputmay be transmitted to an audio amplifier to generate the actuatorsignals 1818. In still other embodiments the broadband carrier signals1828 may be pre-generated and stored in read-only memory in the carriersignal generator 1872, and may be read from memory when the carriersignals 1828 are used by the broadband signal generator 1816.

In one embodiment, the carrier signal generator 1872 generates thebroadband carrier signals 1828 by randomly assigning a phase to each ofthe frequency components. A phase of a frequency component is theposition of a point in time (an instant) on a waveform cycle of thefrequency component. For example, the carrier signal generator 1872 mayrandomly assign a phase to each of the frequency components by randomlyselecting a relative displacement between two corresponding features(for example, peaks or zero crossings) of two frequency components. Thecarrier signal generator 1872 may randomly assign a phase to each of thefrequency components by randomly selecting an initial angle (phaseoffset or phase difference) of a frequency component at its origin. Inanother example, the carrier signal generator 1872 may randomly assign aphase to each of the frequency components by randomly selecting afraction of a wave cycle of a frequency component that has elapsedrelative to the origin.

The actuator signal generator 1876 generates the actuator signals 1818by encoding the parameters 1852 from the sensor signals 1806 into thebroadband carrier signals 1828. The actuator signal generator 1876 maybe implemented in software, hardware, or a combination thereof. Forexample, the actuator signal generator 1876 may include one or moreprocessing units (e.g., a CPU, a GPU, a DSP, a controller, a statemachine, one or more ASICs, one or more RFICs, or any combination ofthese) and a memory.

In one embodiment, the actuator signal generator 1876 generates theactuator signals 1818 by modulating the frequency components using theamplitude of each frequency band 1846. For example, the actuator signalgenerator 1876 may increase or decrease the instantaneous frequency ofthe broadband carrier signals 1828 proportional to the amplitude of eachfrequency band 1846. The broadband carrier signals 1828 are altered suchthat the frequency of the broadband carrier signals 1828 varies in linewith the variations in amplitude of each frequency band 1846. Whenparameters of the sensor signals 1806 are encoded into the broadbandcarrier signals 1828 to generate the actuator signals 1818, the actuatorsignals 1818 increase and decrease in frequency proportional toamplitude of each frequency band 1846. In this way, the frequencies ofthe broadband carrier signals 1828 carry the features of the sensorsignals 1806. The altered broadband carrier signals 1828 are transmittedas actuator signals 1818 to the cutaneous actuators 1822.

In one embodiment, the actuator signal generator 1876 generates theactuator signals 1818 by increasing or decreasing the instantaneousamplitude of the broadband carrier signals 1828 proportional to theamplitude of each frequency band 1846. In one example, parameters of thesensor signals 1806 are superimposed onto the amplitude of the broadbandcarrier signals 1828. The broadband carrier signals 1828 are alteredsuch that the amplitude of the broadband carrier signals 1828 varies inline with the variations in amplitude of each frequency band 1846. Inthis way, the amplitudes of the broadband carrier signals 1828 carry thefeatures of the sensor signals 1806. The altered broadband carriersignals 1828 are transmitted as the actuator signals 1818 to thecutaneous actuators 1822.

In one embodiment, the actuator signal generator 1876 generates theactuator signals 1818 by increasing or decreasing a number of thecutaneous actuators 1822 activated by the actuator signals 1818 based onthe extracted parameters 1852. For example, the actuator signalgenerator 1876 encodes parameters of the sensor signals 1806 into thebroadband carrier signals 1828 by increasing or decreasing a number ofthe cutaneous actuators 1822 activated proportional to the amplitude ofthe sensor signals 1806. As the instantaneous amplitude of the sensorsignals 1806 increases, the broadband carrier signals 1828 are alteredsuch that the actuator signals 1818 activate (turn on) a larger numberof the cutaneous actuators 1822 on the haptic communication device 1820.This may result in a larger area of the user's body 1826 receivingstimulation from the haptic vibrations 1824. This may also create asense of continuous tactile motion (e.g., stroking or moving touch) onthe body 1826 rather than discrete actuation points, as described abovewith reference to FIG. 15B. As the instantaneous amplitude decreases,the broadband carrier signals 1828 are altered such that the actuatorsignals 1818 activate a smaller number of the cutaneous actuators 1822on the haptic communication device 1820. Altering the number ofcutaneous actuators 1822 activated results in more sophisticated hapticcommunication effects. The information related to the words of a socialtouch lexicon described above with reference to FIG. 15B may thuscommunicated by altering the number of the cutaneous actuators 1822activated proportional to the instantaneous amplitude of the sensorsignals 1806. By changing the amplitude and patterns of vibration, alarge number of combinations, rhythms or messages may be reproduced.

In one embodiment, the actuator signal generator 1876 generates theactuator signals 1818 by increasing or decreasing an overlap timebetween haptic vibrations generated by each cutaneous actuator of a pairof the cutaneous actuators 1822 based on the extracted parameters 1852.The overlap time between the beginning of the transmitting of secondhaptic vibrations by a second cutaneous actuator and the end of thetransmitting of first haptic vibrations by a first cutaneous actuatorprovides continuous tactile touch motions instead of the userexperiencing isolated vibrations at different locations on the user'sbody 1826. By changing the overlap time between haptic vibrations, alarge number of combinations, rhythms or messages may be reproduced.

In one embodiment, the actuator signal generator 1876 generates theactuator signals 1818 by increasing or decreasing a duration of thehaptic vibrations 1824 corresponding to the actuator signals 1818 basedon the extracted parameters 1852. A cutaneous actuator 1822 may begintransmitting haptic vibrations 1824 at time t1 and end at time t2 aftera time interval indicating a duration time of the haptic vibrations1824. The smooth continuous motions may be modeled as a function of theduration time t2−t1 of the haptic vibrations 1824 for stroking motion.The duration time t2−t1 determines the speed of the stroke. The durationtime t2−t1 of the haptic vibrations 1824 may be increased or decreasedto provide better communication of the haptic sensations in a messageand words of a social touch lexicon.

In alternative embodiments of the broadband signal generator 1816, thecarrier signal generator 1872 may be located within the actuator signalgenerator 1876. In these embodiments, the sensor signals 1806 may betransmitted by the actuator signal generator 1876 to a PLL section ofthe carrier signal generator 1872 to alter the instantaneous frequencyvalues of the generated broadband carrier signals 1828 to generate theactuator signals 1818. The sensor signals or parameters 1852 may betransmitted by the actuator signal generator 1876 to a level controllersection of the carrier signal generator 1872 to alter the instantaneousamplitude values of the generated broadband carrier signals 1828 togenerate the actuator signals 1818.

The benefits and advantages of the embodiments disclosed herein are asfollows. In general, human evolution has imposed limitations on thespatial and temporal density of transcutaneous information transfer. Ahuman user's skin is optimized to convey information that is transientin nature and requires immediate response, while filtering out stable,nonthreatening stimuli. Precise localization is less important thanquickly drawing attention to the general area of stimulation, so thatthere is not a selective advantage for an organism to develop highspatial resolution. In nature, sustained stimuli are uninformative anddistracting, so the well-known phenomenon of sensory adaptation isbeneficial, but limits the duration that engineered informationconveying stimuli can be delivered without becoming irritating orfiltered out by sensory adaptation. The embodiments disclosed herein usethis knowledge of the skin's response characteristics to generate theactuator signals 1818 for better transmission of the message.

The embodiments disclosed herein filter out the signal features that areoutside the frequency range of skin sensitivity, and focus on the rangeof sensitivity of the skin. The embodiments disclosed herein may furtherpre-filter actuator signals 1818, such that the power is scaledaccording to the frequency-dependence of skin sensitivity, such thatfrequencies with poor responsiveness are amplified and those with goodresponsiveness are attenuated. Because the same signal power isdelivered with lower peak spectral energy, the stimulus is lessirritating and less likely to induce fatigue and sensory adaptation. Inaddition, low spatial resolution results in part from mechanical wavesthat propagate through the tissue. By limiting the peak spectralamplitude, the broadband actuator signals 1818 will have lesspropagation radially from the source compared to narrow-band signalswith equivalent power, thereby improving spatial resolution. Thedisclosed embodiments using broadband actuator signals 1818 thereforemake the artificial and mechanical stimulus feel more natural andphysiologic.

The embodiments disclosed herein also improve the dimensionality of thehaptic communication device 1820. The dimensionality refers to thedifferent independent parameters for encoding information. For example,location, duration, and intensity of stimulation are differentparameters that can be used to encode information. As more dimensions(parameters) are added, more distinct haptic “symbols” are possible,which increases the information transfer. Since broadband signals inducea different quality of perception than narrow-band, they can be used asanother dimension of information encoding. For example, the followingthree states can be used to encode information: sinusoidal waveform with150 Hz frequency, broadband signals with signal power evenly distributedbetween 150 and 350 Hz, and broadband signals with signal power evenlydistributed between 25 and 150 Hz.

FIG. 18D is an illustration of an example process for hapticcommunication using broadband actuator signals 1818, in accordance withan embodiment. In one embodiment, the process of FIG. 18D is performedby the haptic communication system 1800. Other entities (e.g., a remoteconsole or computer) may perform some or all of the steps of the processin other embodiments. Likewise, embodiments may include different and/oradditional steps, or perform the steps in different orders.

The haptic communication system 1800 extracts 1880 parameters 1852 fromsensor signals 1806 describing a message for transmission to a user. Theparameters 1852 may be extracted using the parameter extractor 1842described above. In one embodiment, the parameter extractor 1842 maydecompose the sensor signals 1806 into frequency bands 1846 using anarray of band-pass filters 1844 that span the speech spectrum.

The haptic communication system 1800 generates 1882 broadband carriersignals 1828 by aggregating frequency components. The broadband carriersignals 1828 may include many frequency components including a widerange of frequencies that can be transmitted, recorded, and manipulated.In one embodiment, the broadband carrier signals 1828 may include anumber of frequency components having nearly equal power or amplitude atdifferent frequencies, thus giving the broadband carrier signals 1828 auniform power spectrum 1862.

The haptic communication system 1800 generates 1884 actuator signals1818 by encoding the parameters 1852 from the sensor signals 1806 intothe broadband carrier signals 1828. In one embodiment, the physicallocation on the body 1826 of the user that is being actuated may beencoded into the broadband carrier signals 1828. For example, thelocation of the particular one or more actuators that are driven by theactuator signals 1818 convey information (e.g., identifying a particularphoneme) to the body 1826 of the user. In another embodiment, theamplitude of each frequency component can be used to encode information.In another embodiment, sequences of activation of different actuatorscan be used to encode information. In another embodiment, features ofthe power spectrum 1862 can be used.

The haptic communication system 1800 receives 1886 the actuator signals1818 by cutaneous actuators 1822 communicatively coupled to thebroadband signal generator 1816. In one embodiment, the cutaneousactuators 1822 may be voice coils that convert an analog AC-coupledsignal (the actuator signals 1818) to proportional mechanical motion.The cutaneous actuators 1822 may be arranged on a semi-rigid backingthat secures them in position against the body 1826 with enough pressurethat the haptic vibrations 1824 can be perceived by the user.

The haptic communication system 1800 generates 1888 haptic vibrations1824 corresponding to the actuator signals 1818 on a body 1826 of theuser to communicate the message to the user. A speech message, textualmessage, or a haptic message corresponding to a social touch lexicon maytherefore be transmitted from one user to another user, including hapticmessages for greeting, parting, giving attention, helping, consoling,calming, pleasant, and reassuring touches.

The foregoing description of the embodiments has been presented for thepurpose of illustration; it is not intended to be exhaustive or to limitthe embodiments to the precise forms disclosed. Many modifications andvariations are possible in light of the above disclosure.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope be limited not by this detaileddescription, but rather by any claims that issue on an application basedhereon. Accordingly, the disclosure of the embodiments is intended to beillustrative, but not limiting, of the scope, which is set forth in thefollowing claims.

What is claimed is:
 1. A method, comprising: inputting an audio signalinto a machine learning circuit to compress the audio signal into asequence of actuator signals, the machine learning circuit being trainedby: receiving a training set of acoustic signals; pre-processing thetraining set of acoustic signals into pre-processed audio data, thepre-processed audio data including at least a spectrogram; training themachine learning circuit using the pre-processed audio data, the machinelearning circuit having a cost function based on a reconstruction errorand a plurality of constraints, the machine learning circuit generatinga sequence of haptic cues corresponding to the audio input; and whereinthe sequence of haptic cues is transmitted to one or more cutaneousactuators to generate a sequence of haptic outputs.
 2. The method ofclaim 1, wherein the spectrogram represents the magnitude as a logamplitude.
 3. The method of claim 1, wherein the reconstruction error isgenerated by determining a difference between the pre-processed audiodata and an output from the machine learning circuit trained toreconstruct the pre-processed audio data using the sequence of hapticcues generated from the machine learning circuit.
 4. The method of claim1, wherein the set of constraints includes a restriction on a number ofhaptic cues generated by the machine learning circuit for a particulartime slice.
 5. The method of claim 1, wherein the set of constraintsincludes a constraint that increases a value of the cost function inproportion to a number of activated cutaneous actuators indicated by thehaptic cues generated by the machine learning circuit for a particulartime slice.
 6. The method of claim 1, wherein the set of constraintsincludes a constraint that increases a value of the cost function inproportion to a number state changes of cutaneous actuators as indicatedby the haptic cues generated by the machine learning circuit betweentime slices.
 7. The method of claim 1, wherein the set of constraintsincludes a constraint that increases a value of the cost function whenthe haptic cues generated by the machine learning circuit for a timeslice indicates activation of cutaneous actuators from differentpre-specified groups of cutaneous actuators.
 8. A set of coefficientsfor a machine learning algorithm that is generated by: receiving atraining set of acoustic signals; pre-processing the training set ofacoustic signals into pre-processed audio data, the pre-processed audiodata including at least a spectrogram; training the machine learningalgorithm using the pre-processed audio data, the machine learningalgorithm having a cost function based on a reconstruction error and aplurality of constraints, the machine learning algorithm generating asequence of haptic cues corresponding to the audio input.
 9. The set ofcoefficients of claim 8, wherein the spectrogram represents themagnitude as a log amplitude.
 10. The set of coefficients of claim 8,wherein the reconstruction error is generated by determining adifference between the pre-processed audio data and an output from themachine learning algorithm trained to reconstruct the pre-processedaudio data using the sequence of haptic cues generated from the machinelearning algorithm.
 11. The set of coefficients of claim 8, wherein theconstraints includes a restriction on the number of haptic cuesgenerated by the machine learning algorithm for a particular time slice.12. The set of coefficients of claim 8, wherein the constraints includesa constraint that increases a value of the cost function in proportionto a number of activated cutaneous actuators indicated by the hapticcues generated by the machine learning algorithm for a particular timeslice.
 13. The set of coefficients of claim 8, wherein the constraintsincludes a constraint that increases a value of the cost function inproportion to a number state changes of cutaneous actuators as indicatedby the haptic cues generated by the machine learning algorithm betweentime slices.
 14. The set of coefficients of claim 8, wherein theconstraints includes a constraint that increases a value of the costfunction when the haptic cues generated by the machine learningalgorithm for a time slice indicates activation of cutaneous actuatorsfrom different pre-specified groups of cutaneous actuators.
 15. Anon-transitory computer readable storage medium, comprisinginstructions, that when executed by a processor, cause the processor to:input an audio signal into a machine learning circuit to compress theaudio signal into a sequence of actuator signals, the machine learningcircuit being trained by: receiving a training set of acoustic signals;pre-processing the training set of acoustic signals into pre-processedaudio data, the pre-processed audio data including at least aspectrogram; training the machine learning circuit using thepre-processed audio data, the machine learning circuit having a costfunction based on a reconstruction error and a plurality of constraints,the machine learning circuit generating a sequence of haptic cuescorresponding to the audio input; and wherein the sequence of hapticcues is transmitted to one or more cutaneous actuators to generate asequence of haptic outputs, the one or more cutaneous actuators facing askin surface of a user's body.
 16. The non-transitory computer readablestorage medium of claim 15, wherein the reconstruction error isgenerated by determining a difference between the pre-processed audiodata and an output from the machine learning circuit trained toreconstruct the pre-processed audio data using the sequence of hapticcues generated from the machine learning circuit.
 17. The non-transitorycomputer readable storage medium of claim 15, wherein the constraintsincludes a restriction on the number of haptic cues generated by themachine learning circuit for a particular time slice.
 18. Thenon-transitory computer readable storage medium of claim 15, wherein theconstraints includes a constraint that increases a value of the costfunction in proportion to a number of activated cutaneous actuatorsindicated by the haptic cues generated by the machine learning circuitfor a particular time slice.
 19. The non-transitory computer readablestorage medium of claim 15, wherein the constraints includes aconstraint that increases a value of the cost function in proportion toa number state changes of cutaneous actuators as indicated by the hapticcues generated by the machine learning circuit between time slices. 20.The non-transitory computer readable storage medium of claim 15, whereinthe constraints includes a constraint that increases a value of the costfunction when the haptic cues generated by the machine learning circuitfor a time slice indicates activation of cutaneous actuators fromdifferent pre-specified groups of cutaneous actuators.